This manuscript describes UMSNH-INFOTEC's participation in the first Sentiment Analysis in Dravidian Code-Mixed text task on FIRE 2020. Our solution combines several models that solve the task separately; we then construct a final decision through diferential evolution and the linear combination of models' independently computed decision-values. The generic text categorization systemTC achieves the best performance when a single model is used, and our combined output improves individual performances.
One of the most common tasks involving natural language processing is the so-called Sentiment Analysis (SA). The main objective is to identify the feelings/intentions from a given text. The primary task is identifying the polarity of a text, i.e., if it is positive, negative, or neutral. Despite this simple definition, the task becomes challenging due to small context, errors, negations sentences, polysemy, and figurative language, among other language characteristics. From a machine learning perspective, the first step is to use a text model to transform messages into a vector space, and then, the points in this vector space are used to train a classification model for some specific task. In addition to the labels, the text model contains most of the language and domain knowledge that yields a successful classification.
Text models range from representing each message from sparse word frequency-based to semantic embeddings obtained by deep neural network-based approaches. While frequencybased are fitted only using the inputs text for the task, the word embeddings can be pre-trained over a large text collection. Embedding models can be fit for some specific dataset; this process
Training Development
Test Mixed_feelings
Negative
Positive not-tamil
unknown_state requires a high amount of data and time, and therefore many times, embedding models are used as text transformers.
Even though there is a wide variety of pre-trained embeddings for many languages, the ones with few speakers account with limited resources. In this context, the frequency-based model exhibits a competitive performance against deep learning-based approaches. For our solution, we include multiple embeddings from the flair library [1].
The rest of the paper is organized as follow: first, the Dravidian Code-Mixed tasks are described briefly, at Sections 2. In Sections 3 and 4 basic ideas and models used for our approach are introduced. In Section5, the core of our method is described. Section6 discuss our results at the development phase; the final ranks are also presented. Finally, at Sectio8nsome conclusions are exposed.
The task is a sentiment analysis problem. This task has the particularity that code-mixed texts are part of the corpus data. In this context, the code-mixing term is used to defined texts which are non fully written in its native scripts. The task consists of two sentiment polarity problems for texts written in Malayalam-English and Tamil-English. The process of generating both corpora is introduced in [2, 3, 4, 5, 6].
Both Malayalam and Tamil datasets are split into training, development, and test collections. Messages are labeled with five polarity levelMs: ixed_feelings, Negative, Positive, not-malayalam and unknown_state. Table1 shows the label’s distribution for Malayalam dataset; Ta2ble describes the same information for Tamil dataset.
At its basis, a SA task may be posed as a classification problem. For this case, a set of textsand a set of labelsΘ are given as input to fit model capable of predicting the class for new input texts. The classification problem can be tackled inagnostic way by learning the relationship between the input and the outputΘ i.e. the model (ℒ ( )) = where each label ∈ Θ is the class label (output) for the ∈ is only learned from the input set. Such a model works over a transformed , where the transformation is performed byℒ. The possible text transformations are vast; therefore, they should be adapted for a specific task, input data, and classification strategy. The selection of the best transformation can be performed by hyper-parameter optimization. A state-of-the-art language-independent approach isTC [7]. TC optimizes ℒ by exploring a configuration space comprised of diferent text transformations, tokenizers, and weighting schemes. The exploration is lead by a meta-heuristic and aims to produce efective configuration for Support Vector Machine Classifiers (SVM). As our knowledge about Dravidian dialects is limited, and the input data are in multiple languages, a language-independent approach is best suited for the proposed solution. Our solution aims to integrate a language-independent model enriched by pre-trained embedding models.
For convenience we assume that the output (ℒ ( )) = [ 1, 2, ..., ] is a decision function in a vector form of size , where is the number of diferent labels in Θ. Then, each is the score for belonging to clas s and the class for is the where is maximum.
Since its introduction by Tomas Mikolov 8[], pre-trained word embeddings are often used to train models when there are not large amounts of data for a given task. Even though classic embeddings have limited capacity on learning rare words, Character embedding9s],[ and Pair-Byte embeddings [10] come to alleviate this situation. While embeddings like Word2Ve8c][ and Glove[11] learn from prefix/sufix at the sub-word level, character-based embeddings learn representations specific to the task and domain. On the other hand, Byte-Pair embeddings use a variable-length encoding to iteratively merge the most frequent symbols pairs into a new symbol. This strategy makes both of them a suitable alternative to handle words that are not part of the input vocabulary and small training corpus.
Our model is comprised of an optimized language-independent modelTC with pre-trained embeddings specific for Tamil, Malayalam, and multi-language embeddings. All models are ensembled by a linear combination of SVM classifier decision functions for each text transformations. The weights for each model are adjusted by using diferential evolution. The model is represented at Eq.1.
Θ = 1 (ℒ 1( )) + 1 (ℒ 2( )) … (ℒ ( )),
(
The previous formulation allows us to state our solution as a constrained optimization problem, where coeficients must be optimized to maximize a given fitness function. The weighted score score is maximized for this problem; Eq. 2 defines the optimization problem. maximize score(Θ, Θ )
subject to 0 ≤ ≤ 1.
(
We carry out experiments on five models and combinations: the optimized TC, Byte-Pair embeddings for Multi-language (BP-Multi), language-specific Byte-Pair embeddings, and Character Embeddings. The model selected for submission was the one exhibiting the best scores for the evaluated metrics (weighted versions of score, Precision, and Recall). Tabl3eshows the performances on the Tamil-English development dataset. As can be seen, the best single model performance is achieved by the TC model, while the linear combination increases close to 2%, in our metrics.
On the other hand, Table4 shows the performance for each single text model and the proposed linear combination on the Malayalam-English development dataset. Again the best model performance is achieved by TC, but for the Malayalam, the increase achieved by the linear combination is lower than for the Tamil-English task.
Table5 shows the values for each one of the vector representation when the ensemble model is optimized, see Eq. 2. For both cases, the model in which the highest contribution is TC; however, its contribution is greater for the Malayalam dialect. The second with the higher weight is corresponding to the staked model using Character and Byte-Pair for multi-language embeddings. On the other hand, for Tamil dialect, the weights the Char/BP-Multi is the model with the lowest contribution while stacking Byte-Pair Tamil and multi-language embeddings contribute with the second-highest value. 7. A brief review of TC best model’s parameters For the sake of completeness, Table6 shows the parameters for the best model obtained by the TC system. We briefly summarize the involved parameters described in [12]. Parameters in Table6 may be roughly divided into preprocessing and weighting schemes.
Parameters withhandler sufix (i.e. emoji, hashtag, number, url, and user) have three possible options: delete, group and identity. Thesedelete removes occurrences of entities of the specified kind whilegroup option will change occurrences by a common identifier of the kind. For instance, setting emoji-handler asdelete will remove all emojis in the text. In contrasgt,roup option indicates that emoji’s occurrences must be replaced with the special toke_nemo; this operation is designed for tasks that take advantage of the syntactic information of the token, regardless of the precise value. The identity operator leaves the instances untouched. On the other hand, binary parameters like diacritic-removal, duplication-removal, and punctuationremoval instruct if symbols must be removed or not. The lower-case parameter establishes whether the text is lower case or maintained as it is.
Furthermore, TC allows the use of three diferent tokenizers: • Words -grams. This scheme tokenizes text into words and then produces all possible sub-sequences of words (i.e., − +1 tokens for a text with words). For this parameter, Malayalam’s best performance model includes tokens of length 2, 3, 5, and 9. On the other hand, the Tamil model uses tokens of 1 and 3 words. • Sentences -grams. This approach produces -grams at the character level, i.e., each token is a sub-string of size . Here, the best model for Tamil dialect has tokens of length 2 and Parameter name lower-case emojis-handler hashtag-handlers url-handler user-handler number-handler diacritic-removal duplication-removal punctuation-removal -grams -grams skip-grams weighting scheme token-max-filter token-min-filter
Dravidian dialect Malayalam Tamil
True False identity group identity delete group group group group delete identity
True True
False True
True True
3, 2, 1 3, 2
2, 3, 5, 9 1, 3
none (
1 1 −1 −1
3. In contrast, the Malayalam includes sequences of 1, 2, and 3 characters.
• Skip-grams are word -grams that skip middle parts in words sub-sequences. For this
tokenizer, it must specify the sub-sequence length and the number of middle-words to
skip. This class of tokens is not used for the Malayalam best configuration, but for the
Tamil best model from continuous word sequences of length three, the middle word is
removed, obtaining (
Two frequency-based weighting schemes are used over vector space for the bag of word representations obtained using those mentioned above preprocessing and tokenization strategies. Any of term frequency (TF) and term frequency-inverse document frequency (TFIDF) are selected along with at frequency thresholds using token-min-filter , and token-max-filter . Where all tokens do not reaching the frequency freq or having frequency greater than max-freq are delete. The termmax-freq stands for the frequency of the most repeated token in the collection. Table6 indicates that no token filtering is applied to the vocabularies.
We decided to submit the linear combination based on its development dataset’s performance. Table7 shows the rank for our approach for each task. In both cases, our method was ranked above-average performance.
Task name
Tamil-English Malayalam-English
Precision Recall score Rank
This manuscript presents a model solution for the Dravidian code-mixed text task that integrates language-specific knowledge from pre-trained models with a language-independent model. Our approach is based on the linear combination of several independently created models using diferential evolution. Our proposal can be implemented with open-source libraries, using just a relatively few code lines, and achieves competitive performances for the evaluated tasks. The scripts and data used to implement our approach are availableGaithub1. 1https://github.com/kyriox/dravidian-codemixed [6] B. R. Chakravarthi, Leveraging orthographic information to improve machine translation of under-resourced languages, Ph.D. thesis, NUI Galway, 2020. [7] E. S. Tellez, D. Moctezuma, S. Miranda-Jiménez, M. Graf, An automated text categorization framework based on hyperparameter optimization, Knowledge-Based Systems 149 (2018) 110–123. URL: https://github.com/INGEOTEC/microtc . [8] T. Mikolov, K. Chen, G. Corrado, J. Dean, L. Sutskever, G. Zweig, word2vec, URL https://code. google. com/p/word2vec 22 (2013). [9] B. Heinzerling, M. Strube, Bpemb: Tokenization-free pre-trained subword embeddings in 275 languages, arXiv preprint arXiv:1710.02187 (2017). [10] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural architectures for named entity recognition, arXiv preprint arXiv:1603.01360 (2016). [11] J. Pennington, R. Socher, C. Manning, Glove: Global Vectors for Word Representation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. doi:1 0 . 3 1 1 5 / v 1 / D 1 4 - 1 1 6 2 . [12] E. Tellez, D. Moctezuma, S. Miranda-Jiménez, M. Graf, An automated text categorization framework based on hyperparameter optimization, Knowledge-Based Systems (2018). doi:1 0 . 1 0 1 6 / j . k n o s y s . 2 0 1 8 . 0 3 . 0 0 3 .