The use of social media has grown rapidly during the past few years, which provides a convenient platform for users to communicate by inserting their native language into English and results in a large amount of code mixed text. This way of communication is not only convenient but also reduces people's knowledge burden. They can express their opinions most simply and easily. However, it is quite dificult for non-native speakers to understand these code mixed texts. Therefore, it is important to analyze the sentiment expressed in these texts. This paper reports on our work in Dravidian- Codemix-Fire 20210. We propose a sentiment analysis mBERT-based model, and use self-attention to assign a weight to the output of the BiLSTM, which further improve the analysis accuracy of the model. We test our model on the data sets released by the organizers, and the performance of our system is very close to the best system in the competition. We achieve the weighted average F1-scores of 0.73 and 0.64 in Malayalam and Tamil languages, respectively, and both rank .2
There is an increasing number of people who will express their opinions on social media, including comments on a new movie, expectations, suggestions for a new product, or the responses to a new policy of the government. People will often use their most familiar native language mixed with English to form code mixed text because this relaxed communication way reduces their burden of language as a cognitive process. This is also why the code mixed is becoming more and more common on social media. But for those who are not the native speakers, fully understanding this code mixed texts can be a headache. Therefore, it is important to provide a systematic approach to fully dig the sentiment information given by the code mixed text.
Sentiment analysis is the task of determining subjective opinions or responses about a given
topic. For the past two decades, it has been an active area of academic and industrial research.
There is a growing need for sentiment analysis on social media code mixed tex1t][. Code
mixing is a common phenomenon in the multilingual community and the code mixed text is
sometimes written in non-native scripts. Systems that train on monolingual data fail on code
mixed data because of the complexity of switching code between diferent language levels in
text [
This common task proposes a new gold standard corpus for sentiment analysis of code mixed text in Dravidian (Malayalam-English and Tamil-English). This is the message level polarity classification task. Given a YouTube comment, the system must categorize it into negative, notMalayalam (or not-Tamil), positive, unknown-state or mixed-feeling. We introduce a sentiment analysis mBERT-based model (mBERT means BERT multilingual model) and use self-attention to assign a weight to the output of the BiLSTM, which further improve the analysis accuracy of the model. We also test our model on the data sets released by the organizers, and the performance of our system is very close to the best system in the competition. We achieve the weighted average F1-scores of 0.73 and 0.64 in Malayalam and Tamil languages, respectively, and both rank 2 . Our code is available on GitHu1b
Code mixed text sentiment analysis seminal work was done by Joshi et al3.][on Hindi text. The
sub-word level representations in LSTM (Sub word-LSTM) architecture have been proposed by
Prabhu et al. 4[] on Hindi code mixed text. Shalini et al.5[] created the Kannada-English code
mixed corpus and provided distributed representation methods. Jhanwar et al.6][ combined an
ensemble model of character-n based LSTM and word-n based MBN to identify the sentiment of
Hindi-English code mixed data. The multilayer perceptron model has been used to identify the
polarity of the Bengali-English tweets by Ghosh et a7l].[ Vijay et al. 8[] proposed a supervised
classification system that used a variety of machine learning techniques to detect sentiment.
Lal et al. 9[] proposed a hybrid architecture for the sentiment analysis task in English-Hindi
code mixed data. In [
As mentioned above, there has been a lot of research on sentiment analysis in many diferent
code mixed types of languages. 1[
In this section, after a lot of experiments and comparisons that we choose the pre-trained
methods. Which is the mBERT-based model (mBERT means BERT2 multilingual model)1[
The principle of weight attention
Formally, let C be the set of characters and T be the set of input statements. The sentence s respectively. Concatenating⃖ℎ⃖⃗ and⃖ℎ⃖⃖ obtains the annotation : ∈T can be made of characters by [1, ..., l] where l is the length of input, and thel = 60 in our model. The ℎ forward and backward hidden states of the BiLSTM are represente⃖ℎ⃖d⃗ and⃖ℎ⃖⃖, The attention weight of in the sentence s can be calculated by this following formula: = [⃖ℎ⃖⃗ ⃖⃖⃖], ( = 1, 2, ..., l) ; ℎ
= exp( ) = ∑ exp( )
=1 = =1 h = ∑ And then we get the representation vector h by combination all the weighted outputs: (1) (2) (3) (4) (5)
Next, we do a reshape forh that let it keep the same dimensions with the − output of mBERT. Finally, we concatenate them into the classifier. The schematic overview of the model architecture was shown in a of Figure 1.
The aim of this task is to identify the sentiment polarity of a code mixed dataset of comments/posts in Dravidian (Malayalam and Tamil) collected from social media. Comments/posts may contain more than one sentence, but the average sentence length of the corpus is 1. The datasets consist of YouTube comments labeled into one of the five classes:
Negative(Neg): Tweets contain obvious emotions such as sadness, dissatisfaction, loss, or ofensive language. To express disgust or criticize some people.
Positive(Pos): A tweet that expresses happiness, satisfaction, praise for a person, group or country.
Not-Malayalam or Not-Tamil(Not): For the language of Malayalam or Tamil, if the sentence does not include either Malayalam or Tamil, then it is not Malayalam or Tamil.
Unknown-state(Unknown): Tweets that represent facts, provide news, or belong to advertisements. There’s no obvious emotional expression.
Mixed-feeling(Mixed): Tweets explicitly or implicitly contain the user’s emotions.
Most datas given are written in Roman script, which have the mixture of these forms of
code-mixed sentences –Inter-Sentential switch, Intra-Sentential switch and Tag switching. The
specific data set details are given in table 1. The various types of data are unbalanced from
the table and more details about the dataset can be found i1n4[] and [
In this work, we use mBERT-based as our pre-training model. Before the training, we randomly shufle the data and remove unwanted characters and emoticons (Generally, emoticons express specific sentiment, and we will consider introducing an emoticons system in our future work ). Especially, the label of categorical sentiment values we encode as 0,1,2,3,4 to negative, notTamil(not-Malayalam), positive, unknown or mixed-feelings, respectively. This way is to give a numeric representation to the categorical sentiment data. Finally, we input the processed data into our model. And here mBERT uses the WordPiece3 tool [17] for word segmentation and inserts special separators ([CLS], which separates each sample) and separator ([SEP], which separates diferent sentences in the sample) [18].
Here our model is implemented based on Pytorch. We use Adam optimizer with a learning rate of 1e-5 andCross-Entropy Loss. The batch size is set to 8 and thegradient accumulation steps is set to 4. The epochs and maximum length of the sentence are 5 and 60, respectively.
The measurement to evaluate the participating system is the weighted average F1-scores. Table 2 shows the precision, recall and the weighted average F1-scores of our system. The final ranking is based on the weighted average F1-scores and our submit system obtains the weighted average F1-scores of 0.73 and 0.64 in Malayalam and Tamil languages, respectively.
With the increase of the social media text in popularity and influence, it is increasingly important to analyze the sentiment attached to the text. In this paper, we perform a sentiment analysis on Dravidian Languages (Malayalam-English and Tamil-English). We propose an mBERT-based multilingual processing model. We also give the weights for the hidden state output of the BiLSTM, and then we concatenate it with the original output of the mBERT. Our system achieves very satisfying performance.
On the whole, the article uses mBERT to represent the code-mixed Dravidian text, which has been feed to the BiLSTM (creates attention weighted vector representation of the vector). In the end, the output of BiLSTM and attention layer of mBERT are concatenated for the classification. This framework is well-combined with the help of both high level and low level. Through this model, rich sentiment information is extracted to improve the classification accuracy. In future work, we will consider incorporating emotional information into the classification system. On the other hand, we consider the depth of the model as far as the data allows.
I would like to thank the organizers for their hard work and the teachers for their help. Finally, I would like to thank the school for its support to my research and the future reviewers for their patient work.
3https://github.com/lovit/WordPieceModel Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URLh:ttps://www.aclweb.org/anthology/ 2020.sltu-1.25. [16] B. R. Chakravarthi, Leveraging orthographic information to improve machine translation of under-resourced languages, Ph.D. thesis, NUI Galway, 2020. [17] B. C. Xu, L. Ma, L. Zhang, H. H. Li, M. C. Zhou, An adaptive wordpiece language model for learning chinese word embeddings, in: 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 2019. [18] J. Devlin, M. W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding (2018).