Sentiment analysis of Dravidian languages has received attention in recent years. However, most social media text is code-mixed, and there is no research available on the sentiment analysis of code-mixed Dravidian languages. The Dravidian-CodeMix-FIRE 2020 https://dravidian-codemix.github.io/2020/, a track on Sentiment Analysis for Dravidian Languages in Code-Mixed Text, focused on creating a platform for researchers to come together and investigate the problem. Two language tracks, Tamil and Malayalam, were created as a part of Dravidian-CodeMix-FIRE 2020. The goal of this shared task was to identify the sentiment of a given code-mixed comment (from YouTube) into five classes - positive, negative, neutral, mixed-feeling and comment not in the intended language. The performance of the systems (developed by participants) has been evaluated in terms of weighted-F1 score.
for sentiment analysis of code-mixed text in Dravidian languages (Tamil-English and MalayalamEnglish).
Tamil is one of the Dravidian languages spoken by Tamil people in India, Sri Lanka and by the Tamil diaspora around the world, with oficial recognition in India, Sri Lanka and Singapore. Malayalam is another Dravidian language spoken in the southern region of India with oficial recognition in the Indian state of Kerala and the Union Territories of Lakshadweep and Puducherry [4, 5]. There are nearly 75 million Tamil speakers 1 and 45 million Malayalam speakers 2 in India and other countries. Tamil and Malayalam are highly agglutinative languages [6, 7].
Tamil script evolved from the Tamili script3, Vatteluttu alphabet, and Chola-Pallava script. The modern Tamil script descended from the Chola-Pallava script. It has 12 vowels, 18 consonants, and 1 a¯ytam (voiceless velar fricative). Minority languages such as Saurashtra, Badaga, Irula, and Paniya are also written in the Tamil script [8, 9]. The Malayalam script is the Vatteluttu alphabet extended with symbols from the Grantha script. Both Tamil and Malayalam scripts are alpha-syllabic, belonging to a family of the abugida writing systems that are partially alphabetic and partially syllable-based [10, 11, 12]. However, social media users often adopt Roman script for typing as it is easy to input. Hence, the majority of the data available in social media for these under-resourced languages are code-mixed.
The goal of this task is to identify the sentiment polarity of the code-mixed dataset of comments/posts in Dravidian Languages (Malayalam-English and Tamil-English) collected from social media. The comment/post may contain more than one sentence but the average sentence length of the corpora is 1. Each comment/post is annotated with sentiment polarity at the comment/post level. This dataset also has the class imbalance problem which is consistent with how sentiments are expressed in the real world. The dataset provided for training and development contains 11,335 and 1,260 sentences for Tamil, 4,851 and 541 sentences for Malayalam. More details about the annotation of the dataset can be found in [13] and [14].
Our shared task aims to encourage research that will reveal how sentiment is expressed in codemixed scenarios on Dravidian social media text. The participants were provided with development, training and test datasets.
The Dravidian-CodeMix-FIRE 2020 was a message-level polarity classification task. As a part of this shared task, participants were supposed to develop systems that classify a given Youtube comment into one of the five classes: positive, negative, neutral, mixed emotions, and not in the targeted languages (Tamil or Malayalam). Our datasets consist switching at three levels - Inter-Sentential, Intra-Sentential and Tag. All comments in the dataset were written in Roman script with either Tamil grammar and English lexicon or English grammar and Tamil lexicon. The following examples from the Tamil dataset illustrate this scripting pattern.
• Intha padam vantha piragu yellarum Thala ya kondaduvanga. - After the movie release, everybody will celebrate the hero. Tamil words written in Roman script with no English switch. • Trailer late ah parthavanga like podunga. - Those who watched the trailer late, please like it. Tag switching with English words. 1Between 2011- 2015 Source https://en.wikipedia.org/wiki/Tamil_language 2Between 2011- 2019 Source https://en.wikipedia.org/wiki/Malayalam 3This was also called Damili or Tamil-Brahmi script • Omg .. use head phones. Enna bgm da saami .. - OMG! Use your headphones. Good Lord,
What a background score! Inter-sentential switch • I think sivakarthickku hero getup set aagala. - I think the hero role does not suit Sivakarthick.
Intra-sentential switch between clauses.
The following examples from the Malayalam dataset also show a similar scripting pattern. • Orupaadu nalukalku shesham aanu ithupoloru padam eranghunnathu. - A movie like this is coming out after a long time. Malayalam words written in Roman script with no English switch. • Malayalam industry ku thriller kshamam illannu kaanichu kodukku anghotu. - Show that there is no shortage for thriller movies in Malayalam film industry. Tag switching with English words. • Manju chechiyude athyugran performancenayi kaathirikunnu. The Lady superstar of Malayalam industry. - Waiting for the awesome performance of Manju sister. The Lady superstar of Malayalam film industry. Inter-sentential switch • Next movie ready for nammude swantham dhanush. - Next movie ready for our dear
Dhanush. Intra-sentential switch between clauses.
The data was annotated for sentiments according to the following schema.
• Positive state: There is an explicit or implicit clue in the text suggesting that the speaker is in a positive state, i.e., happy, admiring, relaxed, and forgiving. • Negative state: There is an explicit or implicit clue in the text suggesting that the speaker is in a negative state, i.e., sad, angry, anxious, and violent. • Mixed feelings: There is an explicit or implicit clue in the text suggesting that the speaker is experiencing both positive and negative feeling: Comparing two movies • Neutral state: There is no explicit or implicit indicator of the speaker’s emotional state: Examples are asking for like or subscription or questions about the release date or movie dialogue.
This state can be considered as a neutral state. • Not in intended language: For Malayalam, if the sentence does not contain Malayalam then it is not Malayalam.
The annotators were provided with Tamil and Malayalam translation of the above to facilitate better understanding. Each sentence was annotated by a minimum of three annotators.
We received a total of 32 submissions for Tamil and 28 for Malayalam. The systems were evaluated based on weighted average F1 scores and a rank list was prepared. Table 1 and Table 2 show the rank lists of Tamil and Malayalam tasks respectively. We briefly describe below the methodologies used by the top three teams.
No. TeamName 01 SRJ[15] 02 DT 03 YUN111 [16] 04 codemixed_umsnh [17] 05 LucasHub [18] 06 YNU [19] 07 MUCS[20] 08 PITS [21] 09 datamafia 10 gauravarora [22] 11 jiaming gao 12 Theedhum Nandrum[23] 13 HRS-TECHIE Tam [24] 14 NITP-AI-NLP [25] 15 SSNCSE_NLP[26] 16 zyy1510 [27] 17 bits2020 [28] 18 SSN_NLP_MLRG [29] 19 Siva [30] 20 IRLab@IITV 21 CMSAOne [31] 22 CodeMixedNLP_submission 23 ComMA 24 IRLab@IITBHU Both [32] 25 JUNLP[33] 26 TADS [34] 27 Parameswari_Faith_Nagaruju [35] 28 Judith Jeyafreeda [36] 29 Thirumurugan R 30 DLRG 31 NUIG_Shubhanker [37] 32 Anbukkarasi [38] • SRJ[15]: Authors used XLM-Roberta’s hidden states to extract semantic information. They proposed a new model by extracting the output of top hidden layers in XLM-Roberta and feeding them as inputs to a Convolution Neural Network and finally concatenate them to get better results. They achieved the best result for both Tamil and Malayalam. • YUN111 [16]: This team used mBERT to represent the code-mixed Dravidian text, which has been feed to the BiLSTM (creates attention weighted vector representation of the vector). In the end, the outputs of BiLSTM and attention layer of mBERT are concatenated for the classiifcation. The system achieved Rank 2 for Tamil and Malayalam. • codemixed_umsnh [17]: Authors combined several models that solved the task separately; They then made a final decision through diferential evolution and a linear combination of independently computed decision-value of each model. This system achieved 3rd place for Tamil and • YNU [19]: The system proposed by the team is based on a pre-trained multi-language model XLM-RoBERTa, and uses the K-folding method to the ensemble that aims to solve the sentiment analysis problem of multilingual code-mixed across language models. This system achieved rank 1 for Malayalam and 3 for Tamil.
The distribution of the sentiment classes are imbalanced in both the datasets. In the
MalayalamEnglish code-mixed dataset, we have a class imbalance with the majority of comments belonging
to positive (
Precision =
Recall = +
+
Precision ∗ Recall
F-Score = 2 ∗ Precision + Recall
(
Overall, 119 participants registered for this track. 32 teams submitted final results for Tamil and 28 teams submitted results for Malayalam. Table 1 and Table 2 show the rank list of Tamil and Malayalam task respectively. The runs are sorted in decreasing order of the weighted F1-scores. It is noteworthy that most of the participants used pre-trained embedding such as BERT or its variations even though BERT or its variations are not trained on code-mixed text. Since our corpus contained text written in non-native script, the choice runs counter to linguistic intuitions. There were some systems based on BiLSTM and Recurrent Neural Networks (RNNs). A few other submissions adopted linguistically motivated methods to solve the problem. However they did not achieve good results compared to the BERT based models. Out of all the models proposed, the count vectorization model and the BERTbased model produced the best outcomes. Although there were many systems that were below the baseline results, the approaches taken by participants were diferent and hence we accepted those papers as well in order to encourage diverse research methods to solve the problem. Although weighted scores were considered as the primary metric for our evaluation, it can be noted that class-wise precision, recall, and F1-Score were reported in the most of papers for better understanding of the problem and results.
Some of the participants made interesting observations about the dataset provided by us in and were able to explain the low F1 scores based on that. Although the data was annotated by a minimum of three annotators and an inter-annotator agreement of 0.6 for Tamil and 0.8 for Malayalam in Krippendorf’s alpha was achieved, the dataset contained instances of annotation errors which were pointed out by Krishnamurthy et. al [35] and BalaSundaraRaman et. al [23]. According to Krishnamurthy et al, a few Malayalam sentences belonging to the positive class were wrongly annotated as Not-Malayalam. They also pointed out that some sentences were wrongly tagged as Negative while they actually expressed positive sentiments. According to BalaSundaraRaman et. al [23], there were mismatched predictions in Tamil development dataset, where the authors’ algorithm made correct predictions, but the corresponding manual labels given by the annotators was wrong. To ensure high quality annotation we followed the following protocol. Native language speakers from both the genders were chosen for the task. They were given proper guidelines in their native language and English. We ensured that only after understanding the annotation scheme thoroughly each annotator could proceed to assign the sentiment labels. The manual annotation was carried out in three stages. First, each sentence was annotated by two people. In the second step, the data were collected if both of 4https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html them agreed. In the case of conflict, a third person annotated the sentence. In the third step, if all the three of them did not agree, then two more annotators annotated the sentences. Despite following this strict protocol, errors have occurred in the gold standard dataset. We will consider all these suggestions for next years shared task.
The best performing run achieved weighted F1-score of 0.65 and 0.74 for Tamil and Malayalam respectively. The top team “SRJ" used XLM-Roberta and CNN to propose new model to extract semantic information. These scores are relatively low compared to the monolingual sentiment analysis results in high-resourced languages such as English. Code mixing is a challenging issue since the text is written in non-native script with no standard spelling which causes ambiguities. The word written in non-native script, in our case it is the Latin script, causes variable lexical representations. Since our corpus contains all three types of code-mixing including code-mixing at the word and morpheme levels word-level, it gives rise to out-of-vocabulary problems. Other challenges related to code-mixing are reduplication of words, variations in word order. The challenges faced by the implementations submitted for the shared task reflect the complexity of code-mixing and class imbalance issues in the real-world setting. Coupled with these challenges is the fact that the shared task was conducted in the under-resourced setting which makes it even more dificult to get high results.
This paper overviews the first shared task on sentiment analysis in code-mixed Dravidian text from social media that aims at classifying YouTube comments. The hundred and nineteen teams participated in the task, and a total of 32 teams for Tamil and 28 teams Malayalam submitted the results. Systems have been trained on the unbalanced dataset. The methods proposed by participants ranged from traditional machine learning models with features based approaches to using state-of-the-art embedding methods in deep learning models. In future, we plan to extend the task to other Dravidian languages such as Kannada, Telugu, and Tulu. We also plan to include mixed script data to make the system more real time.
This publication is the outcome of the research supported in part by a research grant from Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289_P2 (Insight_2), co-funded by the European Regional Development Fund as well as by the EU H2020 programme under grant agreements 825182 (Prêt-à-LLOD), and Irish Research Council grant IRCLA/2017/129 (CARDAMOM-Comparative Deep Models of Language for Minority and Historical Languages). proaches to Code Switching, Association for Computational Linguistics, Doha, Qatar, 2014, pp. 116–126. URL: https://www.aclweb.org/anthology/W14-3914. doi:10.3115/v1/W14-3914. [4] T. Dhanabalan, R. Parthasarathi, T. Geetha, Tamil spell checker, in: Sixth Tamil Internet 2003
Conference, Chennai, Tamilnadu, India, 2003. [5] B. Premjith, K. Soman, M. A. Kumar, A deep learning approach for malayalam morphological analysis at character level, Procedia computer science 132 (2018) 47–54. [6] B. R. Chakravarthi, M. Arcan, J. P. McCrae, Improving Wordnets for Under-Resourced Languages Using Machine Translation, in: Proceedings of the 9th Global WordNet Conference, The Global WordNet Conference 2018 Committee, 2018. URL: http://compling.hss.ntu.edu.sg/events/ 2018-gwc/pdfs/GWC2018_paper_16. [7] B. R. Chakravarthi, M. Arcan, J. P. McCrae, WordNet gloss translation for under-resourced languages using multilingual neural machine translation, in: Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, European Association for Machine Translation, Dublin, Ireland, 2019, pp. 1–7. URL: https://www.aclweb.org/anthology/W19-7101. [8] M. Anand Kumar, V. Dhanalakshmi, K. Soman, S. Rajendran, A sequence labeling approach to morphological analyzer for Tamil language, IJCSE) International Journal on Computer Science and Engineering 2 (2010) 1944–195. [9] B. R. Chakravarthi, M. Arcan, J. P. McCrae, Comparison of Diferent Orthographies for Machine Translation of Under-Resourced Dravidian Languages, in: 2nd Conference on Language, Data and Knowledge (LDK 2019), volume 70 of OpenAccess Series in Informatics (OASIcs), Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 2019, pp. 6:1–6:14. URL: http://drops.dagstuhl.de/opus/volltexte/2019/10370. doi:10.4230/OASIcs.LDK.2019.6. [10] B. Krishnamurti, The Dravidian languages, Cambridge University Press, 2003. [11] B. R. Chakravarthi, N. Rajasekaran, M. Arcan, K. McGuinness, N. E.O’Connor, J. P. McCrae, Bilingual lexicon induction across orthographically-distinct under-resourced Dravidian languages, in: Proceedings of the Seventh Workshop on NLP for Similar Languages, Varieties and Dialects, Barcelona, Spain, 2020. [12] B. R. Chakravarthi, Leveraging orthographic information to improve machine translation of under-resourced languages, Ph.D. thesis, NUI Galway, 2020. [13] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www.aclweb.org/anthology/2020.sltu-1. 28. [14] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/2020.sltu-1.25. [15] R. Sun, X. Zhou, SRJ @ Dravidian-CodeMix-FIRE2020:Automatic Classification and Identification Sentiment in Code-mixed Text, in: FIRE (Working Notes), 2020. [16] Y. Zhu, K. Dong, YUN111@Dravidian-CodeMix-FIRE2020: Sentiment Analysis of Dravidian
Code Mixed Text, in: FIRE (Working Notes), 2020. [17] J. Ortiz-Bejar, J. Ortiz-Bejar, J. Cerda-Jacobo, M. Graf, E. S. Tellez, UMSNHINFOTEC@Dravidian-CodeMix-FIRE2020: An ensemble approach based on a multiple text representations, in: FIRE (Working Notes), 2020. [18] B. Huang, Y. Bai, LucasHub@Dravidian-CodeMix-FIRE2020: Sentiment Analysis on Multilingual Code Mixing Text with M-BERT and XLM-RoBERTa, in: FIRE (Working Notes), 2020. [19] X. Ou, H. Li, YNU@Dravidian-CodeMix-FIRE2020: XLM-RoBERTa for Multi-language Sentiment Analysis, in: FIRE (Working Notes), 2020. [20] F. Balouchzahi, H. L. Shashirekha, MUCS@Dravidian-CodeMix-FIRE2020:SACO
SentimentsAnalysis for CodeMix Text, in: FIRE (Working Notes), 2020. [21] N. Kanwar, M. Agarwal, R. K. Mundotiya, PITS@Dravidian-CodeMix-FIRE2020: Traditional
Approach to Noisy Code-Mixed Sentiment Analysis, in: FIRE (Working Notes), 2020. [22] G. Arora, Gauravarora@HASOC-Dravidian-CodeMix- FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection, in: FIRE (Working Notes), 2020. [23] B. L, S. K. Ravindranath, Theedhum Nandrum@Dravidian-CodeMix-FIRE2020: A Sentiment Polarity Classifier for YouTube Comments with Code-switching between Tamil, Malayalam and English, in: FIRE (Working Notes), 2020. [24] S. Swaminathan, H. K. Ganesan, R. Pandiyarajan, HRS-TECHIE@Dravidian-CodeMix-FIRE2020 Social Media Sentiment Analysis for Dravidian Languages using Machine Learning, Deep Learning and Ensemble Approaches, in: FIRE (Working Notes), 2020. [25] A. Kumar, S. Saumya, J. P. Singh, NITP-AI-NLP@Dravidian-CodeMix-FIRE2020: A Hybrid CNN and Bi-LSTM Network for Sentiment Analysis of Dravidian Code-Mixed Social Media Posts, in: FIRE (Working Notes), 2020. [26] N. N. Appiah Balaji, B. B, B. J, SSNCSE_NLP@Dravidian-CodeMix-FIRE2020: Sentiment Analysis for Dravidian Languages in Code-Mixed Text, in: FIRE (Working Notes), 2020. [27] Y. Zhu, X. Zhou, Zyy1510@HASOC-Dravidian-CodeMix-FIRE2020: An Ensemble Model for
Ofensive Language Identification, in: FIRE (Working Notes), 2020. [28] Y. Sharma, A. V. Mandalam, BITS2020@Dravidian-CodeMix-FIRE2020: Sub-Word Level Sentiment Analysis of Dravidian Code Mixed Data, in: FIRE (Working Notes), 2020. [29] A. Kalaivani, D. Thenmozhi, SSN_NLP_MLRG@Dravidian-CodeMix-FIRE2020: Sentiment Code-Mixed Text Classification in Tamil and Malayalam using ULMFiT, in: FIRE (Working Notes), 2020. [30] S. Sai, Y. Sharma, Siva@HASOC-Dravidian-CodeMix-FIRE-2020: Multilingual Ofensive Speech
Detection in Code-mixed and Romanized Text, in: FIRE (Working Notes), 2020. [31] S. Dowlagar, R. Mamidi, CMSAOne@Dravidian-CodeMix-FIRE2020: A Meta Embedding and Transformer model for Code-Mixed Sentiment Analysis on Social Media Text, in: FIRE (Working Notes), 2020. [32] S. Chanda, S. Pal, IRLab@IITBHU@Dravidian-CodeMix-FIRE2020: Sentiment Analysis for Dravidian Languages in Code-Mixed Text, in: FIRE (Working Notes), 2020. [33] S. K. Mahata, D. Das, S. Bandyopadhyay, JUNLP@Dravidian-CodeMix-FIRE2020: Sentiment Classification of Code-Mixed Tweets using Bi-Directional RNN and Language Tags, in: FIRE (Working Notes), 2020. [34] D. Sharma, TADS@Dravidian-CodeMix-FIRE2020: SentimentAnalysisonCodeMixDravidian
Language, in: FIRE (Working Notes), 2020. [35] P. Krishnamurthy, F. Varghese, N. Vuppala, Parameswari_faith_nagaraju@Dravidian-CodeMixFIRE2020: A machine-learning approach using n-grams in sentiment analysis for code-mixed texts:A case study in Tamil and Malayalam, in: FIRE (Working Notes), 2020. [36] J. Jeyafreeda, JudithJeyafreeda@Dravidian-CodeMix-FIRE2020:Sentiment Analysis of YouTube
Comments for DravidianLanguages, in: FIRE (Working Notes), 2020. [37] S. Banerjee, A. Jaypal, S. Thavareesan, NUIG-Shubhanker@Dravidian-CodeMixFIRE2020:Sentiment Analysis of Code-Mixed Dravidian text using XLNet, in: FIRE (Working Notes), 2020. [38] A. S, V. S, SA_SVG@Dravidian-CodeMix-FIRE2020: Deep Learning Based Sentiment Analysis in Code-mixed Tamil-English Text, in: FIRE (Working Notes), 2020. [39] Y. Prakash Babu, R. Eswari, K. Nimmi, CIA_NITT@Dravidian-CodeMix-FIRE2020: MalayalamEnglish Code Mixed Sentiment Analysis Using Sentence BERT And Sentiment Features, in: FIRE (Working Notes), 2020.