This paper describes the approach for Subtask-1 of the FIRE2015 Shared Task on Mixed Script Information Retrieval. The subtask involved multilingual language identi cation (including mixed words and anomalous foreign words), named entity recognition (NER) and subclassi cation. The proposed methodology starts with cleaning the data and then extracting structural and contextual features from the text for further processing. A subset of these features is selected (based on validation) for training supervised classi ers, separately for language identi cation and NER. Finally, they are applied hierarchically to annotate the entire text. The detected named entities are further subclassi ed by a novel unsupervised technique based on query re nement and keyword based scoring. The proposed approach on the testing dataset of the shared task showed promising results with a weighed F-measure of 0.8082. However, it is worth noting that the classi ers have been sub-optimal with respect to discriminating between certain linguistically similar languages (for e.g., Gujarati in Hindi and Gujarati pairs). The proposed approach is exible and robust enough to handle additional languages for identi cation as well as anomalous foreign or extraneous words. The implementation of the approach has also been shared for the purpose of future research usage.
The ubiquitous use of indigenous languages in social media and search engines has led to the increasing need for techniques to tackle mixed script information retrieval. In such an environment, language and named entity identi cation has become both a necessary and challenging task. At this juncture, the shared task for mixed script information retrieval for the FIRE workshop serves as an excellent source to obtain multiple insights to the solution to such a problem. This research paper addresses language identi cation (LI) and Named Entity Recognition (NER) for text in social media. Such kind of text could involve, for example, switching of language in between a sentence (code switching), or even include words of mixed languages (code mixing). In addition to this, people also use phonetic typing and irregular spellings in social media (`2nyt' in place of `tonight' or `y' in place of `why'). All these challenges make the correct identi cation of language in data from social media a tough task.
The task required rigorous preprocessing of both the training (validation) as well as testing dataset. Section 2 describes, in detail, the approach followed for it. Rest of the paper is organized as follows. Section 3 describes all the features extracted. Section 4,5 and 6 describe the hierarchical classi cation approach that we implemented for tagging of tokens into their respective categories. Section 7 and 8 cover the Results and Conclusion. A Web Tool has been developed for the implemented approach and can be accessed at https://mixscian.herokuapp.com. The source code has been shared on the link, https://github.com/ saatvikshah1994/hline for future research purposes. 2.
During training, the following steps were performed to achieve preprocessing: 1. Began preparing the data by skipping all the X's present in the Input le, which are punctuation, numerals, emoticons, mentions (words starting with @), hashtags (words starting with #) and acronyms; with respect to the given annotation le directly. 2. Then Unicode based cleanup of words was performed, to bring Unicode characters to their closest matching ASCII representatives. (Eg. \ZineTM" to ZineTM) 3. After that, each utterance was formatted to remove any punctuation marks in between letters. 4. Finally, erroneous utterances (with unequal number of words and labels) were discarded. During testing, the process started with performing Unicode based cleanup, followed by removal of punctuations or any other invalids in between letters from tokens. 3.
The proposed technique uses a comprehensive set of
features [
AllCaps (if the current word is made up of all capitalized letters), AllSmall (word is constructed with only lowercase characters), WordDigit (word contains digits and alphabets both) InitCaps (whether the rst character is in uppercase) and These features, having variable length for every token under consideration, had to be further processed into a Bag-OfWords formation. Term Frequency Matrix was computed for the same, resulting in the formation of sparse matrices, which were nally passed onto the classi ers. 4.
This section describes the methodology used for tagging Xs, or tags which are punctuation marks, numerals, mentions, hashtags, acronyms or emoticons. The process begin by a rule based approach for isolating the hashtags, mentions, and numerals. This is done by checking for tokens that start with `@', or `#', or tokens that are composed entirely of digits. Then, the tokens are checked for web URLs or email addresses, and punctuations. A regular expression based classi cation was implemented to search for tokens starting with `http://' or `https://', or any token of the format `abc@pqr.xyz'. Also, any tokens that are composed only of punctuation marks are isolated and tagged as X. Another regular expression check is applied to catch any emoticon such as `:-)', `;)', `:g', etc. A dictionary based approach is then applied to check for any token to exist in a list of popular acronyms. If there is a match, the corresponding token is also classi ed as an X. Once these 'X's have been ltered out, a cleaning operation was performed on the remaining tokens. Cleaning is de ned as converting Unicode characters to the nearest matching ASCII characters. For example, converting \ZineTM" to ZineTM. After this preprocessing, these `cleaned' tokens are sent for Named Entity Recognition. 5.
This task focuses on tagging those tokens as named entity
which can be names of persons, locations, organizations etc
[
NE LA, NE OA, NE Ls, NE X, NE XAg Methodology: Tagging of named entities has been divided into two steps:(1)Supervised approach for binary classi cation and (2)Unsupervised approach for sub-classi cation. 5.1
The Supervised approach includes training of Classi ers
[
Composition Features.
An Ensemble Model of NER was developed which performed a "LOGIC OR" on the predicted results of each classi er to decide the nal tag for a particular token. One(1) was used to denote a token being a named entity and Zero(0) for non-named entity.
T agout = 0 OR (T agclassi er 1) OR (T agclassi er 2):::: OR (T agclassi er n) (1) where Tagout is the predicted result for each token, Tagclassi er k is the predicted tag by classi er k, and n is number of classi ers used. 5.2
An Unsupervised approach based on parsing Wikipedia
to nd out whether the predicted named entity is a
person, location or organization was employed [
Assuming the named entity to be of one word, let the named entity that is to be sub-classi ed be denoted by a variable token. The following procedure prevails:1. First of all, any of the punctuation marks, numerals or special characters are removed from token. 2. The token is searched using Wikipedia API for python that returns an object containing the information about token extracted from its Wikipedia page. 3. Using the object, all the above-mentioned information is concatenated to form a single string denoted by wiki-extract. For convenience, the information is rst converted into lowercase, and any of the punctuation marks, numerals, Unicode characters, etc. are removed. Each of the words in wiki-extract is lemmatized, and to counter the internet dependence and decrease the processing time, the nal wiki-extract is cached in a le with the corresponding token against it. 4. The reference set keywords are searched in wiki-extract.
The number of times a keyword is spotted in wikiextract will be the score of the subclass (`NE P', `NE L' or `NE O') to which the keyword belongs. 5. After each of the keywords' search is completed, we get a nal score for each subclass. If the di erence between the scores of the subclass with the maximum score and that with the second maximum score is greater than or equal to a con dence threshold (5 in this paper), then token is annotated as the subclass with the maximum score, otherwise it is annotated as `en'. If the nal scores of all the subclasses is zero, it is marked as `en'.
If in the 2nd step, Wikipedia shows disambiguation error, then token is searched on Bing using its API for python named `py bing search'. As long as the data under process belongs to social networking sites including popular topics like politics, movies, etc., the results by Bing will help in nding out the most popular/probable meaning of token out of many ambiguous cases. Steps to be followed are:i Using `py bing search', token appended with the word `Wikipedia' (for e.g. `AAP wikipedia') is searched on Bing so that it returns URLs of Wikipedia pages at top priority. Only rst three URLs are considered. ii Only those URLs that point towards a Wikipedia page except disambiguation pages are considered. iii As soon as the desired URL is obtained, the proper term that would not cause disambiguation error and could replace token is extracted out of that URL. For e.g. if the token was `AAP'. Bing will return a URL `https://en.wikipedia.org/wiki/Aam Aadmi Party' for this token and then the term `Aam Aadmi Party' extracted out from this URL will replace token. iv Now, rest of the steps of searching the new token on
Wikipedia are same as step 2 to step 5 discussed above.
If, among the three URLs, none of them satis es the condition of being a Wikipedia page except disambiguation pages, then under disambiguation error there is an option to fetch the options o ered by Wikipedia disambiguation page. Only rst ve options are considered in this paper. These ve options are concatenated to form wiki-extract and rest of the procedure is same as step 3 to step 5 keeping the con dence threshold zero.
If in the 2nd step, Wikipedia shows page error, then to replace token with a proper term that would not cause page error, Bing search is used again. Hence, the steps to be followed are same as step i to iv. Again, if among the three URLs obtained from `py bing search', none of them satis es the condition of being a Wikipedia page except disambiguation pages, then the token is annotated as `en'.
Apart from the above algorithm, it is constantly sought that if the term succeeding token also comes out to be named
P R F-S 0.878 0.966 0.838 0.958 0.848 0.9 0.097 0.5 0.163 0.817 0.74 0.776 0.575 0.829 0.679 0.394 0.752 0.517 0.705 0.79 0.745 0.937 0.708 0.806 0.431 0.687 0.529 0.961 0.966 0.964 0.368 0.528 0.433 0.722 0.124 0.214 0.2121 0.25 0.229 entity, provided that token is not positioned last in the current utterance, then they are treated as a single entity and then searched on Wikipedia. For e.g. the tokens are `White' `House' which are marked as named entity. If we search `White' on Wikipedia we won't get desired results. But if we search `White House', we may surely get it as a location, and thus both `White' and `House' are annotated as `NE L'. If the tokens are misspelled or ambiguous, then Wikipedia may cause page or disambiguation error. In that case, the combined token is rst searched on Bing to nd out the term that could replace the combined token. If this results in failure in nding out a Wikipedia page except disambiguation pages, then the process is reverted and the two tokens are searched individually again following the procedure discussed earlier.
This section addresses the problem of identi cation of
language of origin of a query term in the code-mixed query [
Composition Features
Parameter optimization was performed using parameter
tuning based on cross-validation. Supervised approach[
For the purpose of training, the datasets were cleaned and tokens annotated as NE or its subcategories and X's were removed. The trained classi er was then applied on testing set to get the prediction for each token.
In this section, results are reported to proposed experiments. To investigate and nd the most e ective methods as described in upper sections, results are initially obtained by development(Training) set and nally, the model was selected using 4-fold cross validation. Thereafter, a run was submitted for Subtask-1. The overall performance was measured in terms of Utterance, Token and NE accuracies along with precision, recall and F-measure scores for individual languages and sub-divided NE. The Average Fmeasure and Weigthed F-measure scores obtained was 0.654 and 0.808 respectively. While generally the Mean Average F-measure, Weigthed F-measure score was 0.5395, 0.6990 and Max Average F-measure, Weigthed F-measure score was 0.6917, 0.8299.
In this paper, a system was described for Subtask-1 i.e. Query Word Labelling in FIRE Shared Task 2015 on Mixed Script Information Retrieval. The validation procedure indicated that the context of a token, its POS tag, its structure and normalization play key roles in both NER and Language identi cation. It is interesting to note that the performance on linguistically dissimilar languages such as English and Bengali has been top notch. On the other hand, performance on similar language pairs such as Hindi and Marathi have invariably lead to the classi er getting confused. The use of Search engines to re ne query terms followed by use of publically available encyclopedias such as Wikipedia for subclassifying named entities is a newly proposed technique that has given promising results.