With the spurt in usage of smart devices, large amounts of unstructured text is generated by numerous social media tools. This text is often lled with stylistic or linguistic variations making the text analytics using traditional machine learning tools to be less e ective. One of the speci c problem in Indian context is to deal with large number of languages used by social media users in their roman form. As part of FIRE-2015 shared task on mixed script information retrieval, we address the problem of word level language identi cation. Our approach consists of a two stage algorithm for language identi cation. First level classi cation is done using sentence level character n-grams and second level consists of word level character n-grams based classi er. This approach e ectively captures the linguistic mode of author in social texting enviroment. The overall weighted F-Score for the run submitted to FIRE Shared task is 0.7692. The sentence level classi cation algorithm which is used in achiving this result has an accuracy of 0.6887. We could further improve the accuracy of sentence level classi er further by 1.6% using additional social media text crawled from other sources. Naive Bayes classi er showed largest improvement (5.5%) in accuracy level by the addition of supplementary tuples. We also observed that using semi-supervised learning algorithm such as Expectation Maximization with Naive Bayes, the accuracy could be improved to 0.7977.
With the proliferation of social tools like twitter, facebook, etc.. large volumes of text is being generated on a daily basis. Traditional machine learning tools used for text analysis such as Named Entity Recognization(NER) or Parts of Speech Tagging or parsing, are dependent on the premise that the text provided for them are in purer form. They achieve their objective using cooccurrence patterns of features. It has been observed by many studies that social media text when fed to such machine learning algorithm, is often plagued by the excessive out of vocabulary words(sparsity of features). The FIRE-2015 shared task1 addresses the language identi cation task as well as Named Entity recognition in the context of Indian social fora, where the number of languages used is more than 10 and all of them share vocabulary excessively.
To understand the complexity of the task we have posed the
primary problem of word-level language identi cation as a
multi-class classi cation using word lists gathered for each
language. These word-lists are obtained using the method
suggested in
As the text segments are typically social media posts, the number of languages with in a text segment can be safely assumed to be 2. Using this corpus level prior knowledge, we built a two-stage classi cation algorithm. The rst stage consists of identi cation of sentence level language. We used character-level n-grams of each of the sentences as training data for building sentence level classi er. We have used 1,2,3,4,5 grams of all the words in the sentence as the features. We divided the input training data into 80-20 splits using 5-fold cross validation. We built a multi-class classi er using softmax, Naive Bayes and Naive Bayes EM and SVM algorithms using the training data. Among these Naive Bayes EM is a semi-supervised learning algorithm, which uses EM algorithm for improving the test data accuracy. In preparing such training data, we have removed the URLs, X, NE tagged words. The 5-fold cross-validation classi cation accuracy are reported in Table 3. We tried varying the number of n-grams to 3 and 4, which has the e ect of depreciating the accuracy 3-6%. The class-wise distribution of documents in the training data is given in Table 2. We get 82% accuracy when we applied the multi-class Logistic regression based classi er trained on the above data. We have experimented with latent Dirichlet based topic model for this with 100 as the number of topics which was not providing accuracy levels beyond 60%. The results of classi cation using training data are as given in table 3.
After identifying the language pair used for writing a
particular posting, we further build binary classi ers for each
of the language pairs namely, bn-en,gu-en,kn-en,hi-en,
mlen,mr-en,ta-en,te-en. We use the words in the table 1 to
build the binary classi ers. We used Logistic regression
based binary classi er which are giving 92-94% accuracy on
training data. The character n-grams (where n is set to 5)
are used as features for the binary classi er. The approach
suggested in
This approach of using word-level binary classi er works well
as long as the length of the words is su ciently long to
capture the n-gram characteristics of the language of our
interest. But, as we see, tweets often contain stylistic variations
which reduce the length of words signin cantly. When the
length of words is below 3, the words do not carry n-grams
representative of target language. To address this problem
we use words with in a window of two length on either side
to make for the sparsity of features of shorter words. This is
also a heuristic approach e ectively used in the other works
such as
Named Entity detection for short text is much harder task,
as word-colocations and POS tagging do not work well with
mixed script.
We have used
The error analysis is not complete without making the classi er further accuracte. In this regard, we have manually tagged the test data sentences to be of one of the languages
Method of our interest mixing with english. The authors are con dent in tagging 6 of these languages, we depended on other resources for distinguishing malayalam and tamil. As this shared task largely focuses on english being the mixing language we had to classify any of the training data sentences into 9 classes. In order to improve this sentence level language identi cation task, we collected tweets of the 8 languages of our interest using seed words of each of these languages. The seed words are chosen in such a way that the resultant tweets retrieved belonged to the mixed script category. We added atleast 500 tweets for each class to make up a total number of labeled sentences (sentence level tags) to be 7656. We compare the accuracies of various learning algorithms using this expanded training dataset. As we can see accuracy has increased signin cantly by 6% for Naive Bayes and 3% for Logistic regression. We report the summary of experiments conducted in Table 5. The second column shows the classi cation accuracy of sentence level language identi cation with the training data provided in FIRE shared task. The third column shows the accuracy results using the expanded training dataset. We can observe that the semi-supervised version of Naive Bayes (Naive Bayes EM ) is superior among all the classi ers. We can also observe that Naive Bayes classi er is bene tted the most (5.5% increase) by the supplementary tuples added to the training data. Naive Bayes EM is improved by approximately 3% and MaxEnt is improved by 1.6%.
In the current study we addressed the language identi cation problem in mixed script socail media text, at the word level, involving multiple indian languages namely bengali, gujarati, hindi, kannada, malayalam, marathi, tamil, telugu. Observing that the social media mixed script posts often involve english as the mixing language and can involve atmost one more other language as the length of the messages are quite short, we used a two-stage classi cation approach for sentence level langauge mode of the author and then a binary classi er for distinguishing english and each of the speci c languages listed above. The test run submitted has given overall weighted F-measure of 0.7692. The sentence level classi cation accuracy was 68.87%. We could further improve this accuracy to 79.77% using abundantly available social media tweets crawled using seed words of speci c langauge.
We sincerely thank the active participation of members of CoE Analytics, IDRBT, especially K. Sai Kiran, B. Shiva Krishna for helping us in sentence level labeling, and word level labeling. We thank IDRBT for providing the research environment for executing this task.