In social media communication, code switching has become quite a common phenomenon especially for multilingual speakers. Automatic language identi cation becomes both a necessary and challenging task in such an environment. In this work, we describe a CRF based system with voting approach for code-mixed query word labeling at word-level as part of our participation in the shared task on Mixed Script Information Retrieval at Forum for Information Retrieval Evaluation (FIRE) in 2015. Our method uses character n-gram, simple lexical features and special character features, and therefore, can easily be replicated across languages. The performance of the system was evaluated against the test sets provided by the FIRE 2015 shared task on mixed script information retrieval. Experimental results show encouraging performance across the language pairs.
ACM proceedings; LATEX; text tagging
Though South and South East Asian languages have their
own indigenous scripts, these languages are mostly written
using Roman script in the social media such as tweets, blogs,
etc., due to various socio-cultural and technological reasons.
The use of Roman script for such languages presents serious
challenges to understanding, search and language identi
cation. Abundant use of Roman script on the Web not only
for documents as well as for user queries to search the
documents needs to be addressed. Although language identi
cation at document level is a well-studied natural language
problem [
ne-grained language identi cation from more than one
language is still very challenging and error prone when the
spans of text are smaller. Unsupervised and supervised
approaches were investigated for the detection of four language
pairs, Spanish-English, Modern Standard Arabic and Arabic
dialects, Chinese-English and Nepalese-English, at the word
level in code-switching data. The results of the task
revealed that language identi cation in code-switching is still
far from solved and warrants further natural language
processing research. Shared tasks on language identi cation
have been organized in FIRE since 2013 and various
attempts [
A query or utterance q : < w1 w2 w3 ... wn > is written in Roman script. The words or tokens, w1,w2,w3 etc., could be standard English (en) words or transliterated from any of the eight Indian languages, namely Bengali (bn), Hindi (hi), Gujrati (gu), Kannada (kn), Malayalam (ml), Marathi (mr), Tamil (ta), Telugu (te) under consideration in this subtask. The main objective of this task is to perform wordlevel language identi cation (WLL), i.e. to label each token with single tag belongs to one of the ve categories shown in Table 1. Though some of the categories have also ner subcategories, the identi cation of such subcategories is not mandatory. 3.
This section describes the training and test dataset that were provided to the task participants by the task organizers. The training dataset was provided in the form of set of sentences and respective tags for each token of the sentences. The training dataset consists of 2908 utterances, whereas the test dataset contains 792. Apart from the dataset provided by the task organizers we did not use any external dataset or resources to either train or ne-tune our system.
An empirical study on the development data reveals the following facts: a) the average length of all the tokens is greater than 5 and b) majority of the tokens belong to the English language. 4.
Our word identi cation process involves three steps- At rst we have independently applied multiple classi ers which have been developed using CRF. Then voting approach has been employed over the outputs of the classi ers which are applied in rst step. Finally, we have employed a classi er which deals with NE and MIX tags. Also we have tackled the con ict situations those come up through voting (discussed in section 4.2). 4.1
We have developed in total nine classi ers. Eight di erent IL(N) where N=BN,GU,HI,KN,ML,MR, TA,TE classication models were built for eight Indian languages (ILs), namely BN-classi er, GU-classi er, HI-classi er, KN-classi er, ML-Classi er, MR-classi er, TA-classi er and TE-classi er. While training a IL(N) classi er, tokens of the type NE, MIX, Others and all other ILs were assigned R tag.The output of IL(N) classi er could be one of the four- i) N ii) X (for punctuation) iii) en (for English) and iv) R (for any IL except N, NE, MIX and Others). Apart from eight IL(N) classi ers, we have trained another classi er (namely ALLclassi er) using all the existing tags in the supplied training dataset. The ALL-classi er has dealt with the NE, MIX and Others tokens as well as served as tie breaker (discussed in section 4.2).
In this work, Conditional Random Field (CRF) has been employed to build all of the classi er models. We used CRF++ toolkit1 which is a simple, customizable, and open source implementation of CRF. All of these nine classi ers used the same set of features listed below in the following subsections. 4.1.1
Recent studies [
A token might either start with some symbol, e.g. #aapstorm, @timesnow or it may contain such symbols within, e.g. a***a, bari-r etc. Sometimes the entire word is built up of a symbol, e.g. ", ?.
has symbol(token) = 4.1.3
This feature was used as a binary feature. If a token is a link, i.e. if it starts with "http://", "https://" or "www." then the value is set to 1, otherwise it is set to 0. if token has symbols otherwise is link(token) = if token is a link otherwise 4.1.4
In case of chat dialogue the use of digit(s) in a word often means di erent than their traditional use. For example, `n8' could mean `night', `2' could mean `to' or `too'. It is also found that most of the cases such words contain numerical digits in single position. Therefore, in our system we have used the presence of single digit in any alphanumeric word as binary feature. has digit(token) = 4.1.5
It is an established fact that any language dependent
feature increases the accuracy of language identi cation
systems for that particular language. Also recent studies on
xed length su x feature had been carried out and were
successfully used by [
has suf f ix(token) =
This case is straight forward, i.e. no con ict between the outputs of all the eight IL(N) classi ers for a single token, meaning all the IL(N) classi ers agree on the tag of that token.
Rule 1: This rule is applicable for only En and X tags. If the output of all the classi ers for a particular token is same and either EN or X, then that particular tag is chosen as the nal tag for the given token. For example, the token #aapsweep is labeled as X by all eight classi ers. Thus, the nal tag of this token becomes X. #aapsweep X X X X X X X X ) X
Rule 2: If all the tags are same but other than EN or X, then we consider the output of the ALL-classi er for the said token as the nal tag. This phenomenon only occurs when all the eight IL(N) classi ers identi ed the token as R. For example, in the following example, the token `saaf' is marked as R by all the eight IL(N) classi ers. Since the label generated by the ALL-classi er for `saaf' is HI, so the nal tag of the token becomes HI.
saaf R R R R R R R ) HI 4.2.2
In this scenario, output of all the classi ers for a given token is limited between two tags. Based on the tags involved in such con icts this situation is further classi ed into subcategories which are discussed in the following subsections.
Rule 3: If con ict is between R and any other language tag including EN, then the tag other than R marked by the classi er is selected as the nal tag of the token. In the following example, the token doctor is marked as either EN or IL by the language classi ers. Therefore, the nal tag of doctor is EN.
doctor R R R R R R EN EN ) EN
Rule 4: If the classi ers di er in between two tags other than R, then a voting is counted in support of each of the two tags. Finally the tag with maximum votes is assigned as the nal tag for the given token. In the example, the no. of votes in favor of EN tag for the token take is greater than the no. of votes supporting BN.
take BN EN EN EN EN EN EN EN ) EN 4.2.3
Rule 5: If the con ict involves a) R, b) EN or X and c) any of the eight Indian Language tags, then we rst replace all the R tags with the other Indian Language tag involved in the con ict, thus reducing the con ict between three tags scenario into con ict between two. Finally Rule 4 is applied to decide the nal tag. For example; ore BN EN R R R R R R
+ ore BN EN BN BN BN BN BN BN ) BN
Rule 6: If the con ict involves three tags and none of those three are R, then simple majority voting was applied to choose the nal tag. 4.2.4
Rule 7: In case there is disagreement between more than three language classi er for a single token, the nal label of that token is decided by the All-classi er. The occurrence of such cases is very rare. 4.3
Since we have not included any feature speci cally to handle the NE or MIX tokens, we have depended entirely on the All-classi er to mark the NE and MIX tokens. So, if a token is marked as NE by the All-classi er then the nal tag of the token becomes NE, irrespective of the outputs of the eight language classi ers for the same token. The same procedure is applied to mask MIX tokens. 5.
In case of language category maximum accuracy is achieved for en tokens, which is 0.7838. Whereas, the accuracy is pretty low for Gujrati, Malayalam and Kannada languages (shown in Table 3). We have dwelled upon the result and observed that it is due to the lower amount of tokens presence in the development set for these three languages. For example, the number of gu tokens present in the development set is only 890, which is very few when compared to that of en tokens, i.e. 17957.
Overall, our system achieved the accuracy (weighted fmeasure) of 0.700373312. Out of 11999 tokens in the testset 8582 tokens were marked correctly. However, as our system didn't consider any contextual information, the accuracy achieved at the utterance level was expectedly very low at 0.128788. Only in 102 occasions all the tokens of an entire utterance was labeled with correct tags. More detailed analysis of the result can be done once the gold standard data is shared by the task organizers.
In this paper, we have presented a brief overview of our hybrid approach to address the automatic WLL identi cation problem. We have observed that the voting approach on multiple classi ers output provides better results than use of a single classi er system. For our participation in Query Word Labeling subtask, we have submitted two runs: the rst one, i.e. Run1 using the system as described above and the other, i.e. Run2 using only the ALL-classi er without the need of any voting mechanism, and the obtained results con rm that the overall accuracy of Run1 is more than 10% higher when compared to Run2.
As future work, we would like to explore more sophisticated features to handle NE or O tags and better postprocessing heuristics for handling MIX tags in the WLL identi cation task and try to improve the performance of system by using context modelling. We also plan to incorporate more language speci c feature in our future work to improve the accuracy of the system.
We acknowledge the support of the Department of Electronics and Information Technology (DeitY), Government of India, through the project \CLIA System Phase III".
The research work of the second last author was carried out in the framework of WIQ-EI IRSES (Grant No. 269180) within the FP 7 Marie Curie, DIANA-APPLICATIONS (TIN201238603-C02-01) projects and the VLC/CAMPUS Microcluster on Multimodal Interaction in Intelligent Systems.