Social media has penetrated into multi-lingual societies, however most of them use English to be a preferred language for communication. So it looks natural for them to mix their cultural language with English during conversations resulting in abundance of multilingual data - cacloldteh-misixed data, available in today's world. Downstream NLP tasks using such data is challenging due to the semantic nature of it being spread across multiple languages. One such NLP task is Sentiment analysis; for this we use an auto-regressive XLNet model to perform sentiment analysis on code-mixed Tamil-English and Malayalam-English datasets.
Multilingual users have the tendency to mix linguistic units in the social media resulting
in code-mixed data being easily available. The phenomenon of code-mixing is explained in
[
Since code-mixed is mostly sourced from social media platforms, the data in it’s raw form is highly unstructured and hence corpus creation to organize this unstructured data into datasets for further analysis pose a challenge. For some of the Indian langua1g5e]sh,a[s compiled a Tamil-English code-mixed dataset, the first annotated Tangl3ishdataset. Similarly,16[] published a dataset for Malayalam-English code-mixed data, where the authors also provided references to the availability of other code-mixed datasets such as Chinese-English and SpanishEnglish. But significant work hasn’t been done in the area of corpus creation for code-mixing for Indian languages. The Indian languages are considered to be under-resourced and so there is less interest in performing NLP tasks on these languages.
Sentiment analysis is a well known NLP task that infers the positive, negative and neutral
sentiments from a statement in question. However there are very few works on the sentiment
analysis over code-mixed data1;7[][18] provides an overview of the work done on sentiment
analysis of Dravidian code-mixed text. Another wor1k9,][compares the performance of
diferent transformer architectures on the task of sentiment analysis of code-mixed da2t0a]. [
employed an approach based on lexicon to assign sentiment to Hindi-English code-mixed text.
[21] illustrates a method to detect hate speech in code-mixed Hinglish dataset. For the purpose
of conducting this research they used FIRE 2013 and FIRE 2014 dataset2s.2][ used a LSTM
[23] based approach to improve the state-of-art performan2c0e] [on the hinglish datasets by
18 percent. [24] used shared parameters in a siamese networ2k5[] to project the code-mixed
sentences and sentences in standard languages into a common sentiment space. The similarity
of projected sentences is an indicative of how similar their sentiments are, similar sentences
have similar sentiment. Ensemble based techniques have also been used for sentiment analysis
of code-mixed data,2[
3Tanglish refers to code switching between Tamil and English, a term predominantly used in the Tamil community
In spoken and written conversations, it is observed that the usage of lexicon, connectives and phrases from English are used in combination with other languages; this can very well be seen in the social media text and in spoken conversations across geographies, especially in India.
Sentiment Analysis in social media has drawn attention in recent years. However, sentiment analysis on Tamil-English (Tanglish) and Malayalam-English code-mixed data are not readily available for research. The authors1o5f] [and [16] have collected 184,573 sentences for Tamil and 116,711 sentences for Malayalam from YouTube comments which are based on the trailers of the movies released in 2019 for building Tamil-English and Malayalam-English datasets where non-code-mixed sentences were removed from the collection. Further, emoticons were removed and sentence length filters were applied to render the mentioned datasets. In the end two data sets of size 15,744 and 6,738 sentences were reported for Tanglish and Malayalam-English texts.
To get this dataset ready for sentiment analys1is5][ and [16] refers to manual annotation activity carried out with three annotators annotating each sentence in the data set. The Krippendorf’s alpha (α) is used to measure inter-annotator agreement which is 0.6585 and 0.890 for Tamil and Malayalam code-mixed data sets respectively.
The dataset was provided for this task in three parts training, validation and testing. The number of sentences used for the dataset splits are provided in ta1b.lBeoth these datasets are released in DravidianCodeMix FIRE 2020 competition organized by dravidiancodemixed. These comments were grouped into five categories positive, negative, neutral, mixed emotions, or not in the intended languages.
Language models have been integral to the recent advances made in the field of NLP due to its ability to predict the next token in a sequence. Traditionally this achieved by computing the joint distribution of the tokens in a sequence as a function of conditional probability distribution of each token given other tokens in the sequence.
However, XLNet[
The experiment results for the experiments outlined in the previous section are provided in table 2. We were able to achieve 0.49 & 0.35 accuracies and 0.52 & 0.32 F-scores on both the datasets respectively. The results are biased towaProdssitive class because of the class-imbalance seen in the training set. Further, it can be seen that the model performs better on the Malayalam-English dataset despite the Tanglish dataset having more samples; this can be attributed to more noise in the Tamil-English data and hence relatively poor performance. Our results do not perform better than the baseline-results described 1in5][and [16]. We hypothesize that these results can further be improved by training the model for more epochs with a pre-processing step performed in combination with oversampling and undersampling of the minority and majority classes respectively. Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. UhRttLp:s://www. aclweb.org/anthology/2020.sltu-1.28 [16] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URhLtt:ps://www.aclweb.org/anthology/ 2020.sltu-1.25. [17] B. R. Chakravarthi, R. Priyadharshini, V. Muralidaran, S. Suryawanshi, N. Jose, E. Sherly, J. P. McCrae, Overview of the track on Sentiment Analysis for Dravidian Languages in Code-Mixed Text, in: Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2020). CEUR Workshop Proceedings. In: CEUR-WS. org, Hyderabad, India, 2020. [18] B. R. Chakravarthi, R. Priyadharshini, V. Muralidaran, S. Suryawanshi, N. Jose, E. Sherly, J. P. McCrae, Overview of the track on Sentiment Analysis for Dravidian Languages in Code-Mixed Text, in: Proceedings of the 12th Forum for Information Retrieval Evaluation, FIRE ’20, 2020. [19] S. Banerjee, B. R. Chakravarthi, J. P. McCrae, Comparison of pretrained embeddings to identify hate speech in Indian code-mixed text, in: 2nd IEEE International Conference on Advances in Computing, Communication Control and Networking –ICACCCN (ICAC3N20), 2020. [20] S. Sharma, P. Srinivas, R. C. Balabantaray, Text normalization of code mix and sentiment analysis, in: 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2015, pp. 1468–1473. [21] P. Rani, S. Suryawanshi, K. Goswami, B. R. Chakravarthi, T. Fransen, J. P. McCrae, A comparative study of diferent state-of-the-art hate speech detection methods in HindiEnglish code-mixed data, in: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, 2020, pp. 42–48. URL: https://www.aclweb.org/anthology/2020.trac-.1.7 [22] A. Prabhu, A. Joshi, M. Shrivastava, V. Varma, Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text (2016). [23] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997) 1735–1780. [24] N. Choudhary, R. Singh, I. Bindlish, M. Shrivastava, Sentiment analysis of code-mixed languages leveraging resource rich languages, 2a0r1X8i.v:1804.00806. [25] S. K. Roy, M. Harandi, R. Nock, R. Hartley, Siamese networks: The tale of two manifolds, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 3046–3055. [26] M. G. Jhanwar, A. Das, An ensemble model for sentiment analysis of hindi-english codemixed data, arXiv preprint arXiv:1806.04450 (2018). [27] S. Ghosh, S. Ghosh, D. Das, Sentiment identification in code-mixed social media text, arXiv preprint arXiv:1707.01184 (2017). [28] A. Kumar, H. Agarwal, K. Bansal, A. Modi, Baksa at semeval-2020 task 9: Bolstering cnn with self-attention for sentiment analysis of code mixed text, 2020. a r X i v : 2 0 0 7 . 1 0 8 1 9