Overview of the Mixed Script Information Retrieval (MSIR) at FIRE-2016

Abstract

The shared task on Mixed Script Information Retrieval (MSIR) was organized for the fourth year in FIRE-2016. The track had two subtasks. Subtask-1 was on question classi cation where questions were in code mixed Bengali-English and Bengali was written in transliterated Roman script. Subtask-2 was on ad-hoc retrieval of Hindi lm song lyrics, movie reviews and astrology documents, where both the queries and documents were in Hindi either written in Devanagari script or in Roman transliterated form. A total of 33 runs were submitted by 9 participating teams, of which 20 runs were for subtask-1 by 7 teams and 13 runs for subtask-2 by 7 teams. The overview presents a comprehensive report of the subtasks, datasets and performances of the submitted runs.

1. INTRODUCTION

A large number of languages, including Arabic, Russian, and most of the South and South East Asian languages like Bengali, Hindi etc., have their own indigenous scripts. However, the websites and the user generated content (such as tweets and blogs) in these languages are written using Roman script due to various socio-cultural and technological reasons[ 1 ]. This process of phonetically representing the words of a language in a non-native script is called transliteration. English being the most popular language of the web, transliteration, especially into the Roman script, is used abundantly on the Web not only for documents, but also for user queries that intend to search for these documents. This situation, where both documents and queries can be in more than one scripts, and the user expectation could be to retrieve documents across scripts is referred to as Mixed Script Information Retrieval.

The MSIR shared task was introduced in 2013 as \Transliterated Search" at FIRE-2013 [13]. Two pilot subtasks on transliterated search were introduced as a part of the FIRE2013 shared task on MSIR. Subtask-1 was on language identi cation of the query words and subsequent back transliteration of the Indian language words. The subtask was conducted for three Indian languages - Hindi, Bengali and Gujarati. Subtask-2 was on ad hoc retrieval of Bollywood song lyrics - one of the most common forms of transliterated search that commercial search engines have to tackle. Five teams participated in the shared task.

In FIRE-2014, the scope of subtask-1 was extended to cover three more South Indian languages - Tamil, Kannada and Malayalam. In subtask-2, (a) queries in Devanagari script, and (b) more natural queries with splitting and joining of words, were introduced. More than 15 teams participated in the 2 subtasks [ 8 ].

Last year (FIRE-2015), the shared task was renamed from \Transliterated Search" to \Mixed Script Information Retrieval (MSIR)" for aligning it to the framework proposed by [ 9 ]. In FIRE-2015, three subtasks were conducted [16]. Subtask-1 was extended further by including more Indic languages, and transliterated text from all the languages were mixed. Subtask-2 was on searching movie dialogues and reviews along with song lyrics. Mixed script question answering (MSQA) was introduced as subtask-3. A total of 10 teams made 24 submissions for subtask-1 and subtask-2. In spite of a signi cant number of registrations, no run was received in subtask-3.

This year, we hosted two subtasks in the MSIR shared task. Subtask-1 was on classifying code-mixed cross-script question; this task was the continuation of last year's subtask3. Here Bengali words were written in Roman transliterated Bengali. Here Bengali words were written in Roman transliterated Bengali. The subtask-2 was on information retrieval of Hindi-English code-mixed tweets. The objective of subtask-2 was to retrieve the top k tweets from a corpus [ 7 ] for a given query consisting of Hind-English terms where the Hindi terms are written in Roman transliterated form.

This paper provides the overview of the MSIR track in the Eighth Forum for Information Retrieval Conference 2016 (FIRE-2016). The track was coordinated jointly by Microsoft Research India, Jadavpur University, Technical University of Valencia, IIIT Sriharikota and NIT Agartala. Details of these tasks can also be found on the website https: //msir2016.github.io/.

The rest of the paper is organized as follows. Section 2 and 3 describe the datasets, present and analyze the run submissions for the Subtask-1 and Subtask-2 respectively. We conclude with a summary in Section 4.

2. SUBTASK-1: CODE-MIXED CROSS

SCRIPT QUESTION ANSWERING

Being a classic application of natural language processing, question answering (QA) has practical applications in various domains such as education, health care, personal assistance, etc. QA is a retrieval task which is more challenging than the task of common search engines because the purpose of QA is to nd accurate and concise answer to a question rather than just retrieving relevant documents containing the answer [ 10 ]. Recently, the code-mixed crossscript QA research problem was formally introduced in [ 3 ]. The rst step of understanding a question is to perform question analysis. Question classi cation is an important task in question analysis which detects the answer type of the question. Question classi cation helps not only lter out a wide range of candidate answers but also determine answer selection strategies [ 10 ]. Furthermore, it has been observed that the performance of question classi cation has signi cant inuence on the overall performance of a QA system.

Let, Q = fq1; q2; : : : ; qng be a set of factoid questions associated with domain D. Each question q : hw1w2w3 : : : wpi, is a set of words where p denotes the total number of words in a question. The words, w1; w2; w3; : : : ; wp, could be English words or transliterated from Bengali in the code mixed scenario. Let C = fc1; c2; : : : ; cmg be the set of question classes. Here n and m refer to the total number of questions and question classes respectively.

The objective of this subtask is to classify each given question qi 2 Q into one of the prede ned coarse-grained classes cj 2 C. For example, the question \last volvo bus kokhon chare?" (English gloss: \When does the last volvo bus depart?") should be classi ed to the class `TEMPORAL'. 2.1

Datasets

We prepared the datasets for subtask-1 from the dataset described in [ 3 ] which is the only dataset available for codemixed cross-script question answering research. The dataset described in [ 3 ] contains questions, messages and answers from the sports and tourism domains in code-mixed crossscript English{Bengali. The dataset contains a total of 20 documents from two domains, namely sports and tourism. There are 10 documents in the sports domain which consist of 116 informal posts and 192 questions, while the 10 documents in the tourism domain consist of 183 informal posts and 314 questions. We initially provided 330 labeled factoid questions as the development set to the participants after accepting the data usage agreement. The testset contains 180 unlabeled factoid questions. Table 1 and Table 2 provide statistics of the dataset. Question class speci c distribution of the datasets is given in Figure 1.

A total of 15 research teams registered for subtask-1. However, only 7 teams submitted runs and a total of 20 runs were received. All the teams submitted 3 runs except AMRITA CEN who submitted 2 runs.

AMRITA CEN [ 2 ] team submitted 2 runs. They used bag-of-words (BoW) model for the Run-1. The Run-2 was based on Recurrent Neural Network (RNN). The initial embedding vector was given to RNN and the output of RNN was fed to logistic regression for training. Overall, the BoW model outperformed the RNN model by almost 7% ons F1measure.

AMRITA-CEN-NLP [ 4 ] team submitted 3 runs. They approached the problem using Vector Space Model (VSM). Weighted term based on the context was applied to overcome the shortcomings of VSM. The proposed approach achieved upto 80% accuracy in terms of F1-measure.

ANUJ [15] also submitted 3 runs. The author used term frequency ^aAS inverse document frequency (TF-IDF) vector as feature. A number of machine learning algorithms, namely Support Vector Machines (SVM), Logistic Regression (LR), Random Forest (RF) and Gradient Boosting were applied using Grid Search to come up with the best parameters and model. The RF model performed the best among the 3 runs.

BITS PILANI [ 5 ] submitted 3 runs. Instead of applying the classi ers on the code-mixed cross-script data, they convert the data into English. The translation was performed using Google translation API 1. Then they applied three machine learning classi ers for each run, namely Gaussian Nave Bayes, LR and RF Classi er. However, Gaussian Naive Bayes classi er outperformed the other two classi ers.

IINTU [ 6 ] was the best performing team. The team submitted 3 runs which were based on machine learning approaches. They trained three separate classi ers namely RF, One-vs-Rest and k-NN, followed by building an ensemble classi er using these 3 classi ers for the classi cation task. The ensemble classi er took the output label by each of the 1https://translate.google.com/ individual classi ers and selected the majority label as output. In case of tie any one label was chosen at random as output.

NLP-NITMZ [ 12 ] submitted 3 runs of which 2 runs were rule based - a rst set of direct rules were applied for the Run-1 while a second set of dependent rules were used for the Run-3. A total of 39 rules were identi ed for the rule based runs. Nave Bayes classi er was used in Run-2 whereas Nave Bayes updateable classi er was used in Run-3.

IIT(ISM)D used three di erent machine learning based classi cation models - Sequential Minimal Optimization, Nave Bayes Multimodel and Decision Tree FT to annotate the question text. This team submitted the runs after the deadline. 2.3

Results

In this section, we de ne the evaluation metrics used to evaluate the runs submitted to the subtask-1. Typically, the performance of a question classi er is measured by calculating the accuracy of that classi er on a particular test set [ 10 ]. We also used this metric for evaluating the code-mixed cross-script question classi cation performance.

accuracy = number of correctly classi ed samples

total number of testset samples

In addition, we also computed the standard precision, recall and F1-measure to evaluate the class speci c performances of the participating systems. The precision, recall and F1-measure of a classi er on a particular class c are de ned as follows: number of samples correctly classi ed as c

number of samples classi ed as c precision(P ) = recall(R) = number of samples correctly classi ed as c

total number of samples in class c F 1 measure = 2:P:R P + R

Table 3 presents the performance of the submitted runs in terms of accuracy. Class speci c performances are reported in Table 4. A baseline system was also developed for the sake of comparison using the BoW which obtained 79.444% accuracy. It can be observed from Table 3 that the highest accuracy (83.333%) was achieved by the IINTU team. The classi cation performance on the temporal (TEMP) class was very high for almost all the teams. However, Table 4 and Figure 2 suggest that the miscellaneous (MISC) question class was very di cult to identify. Most of the teams could not identify the MISC class. The reason could be very low presence(2%) of MISC class in the training dataset.

3. SUBTASK-2:INFORMATION RETRIEV

AL ON CODE-MIXED HINDI-ENGLISH

TWEETS

This subtask is based on the concepts discussed in [ 9 ]. In this subtask, the objective was to retrieve Code-Mixed Hindi-English tweets from a corpus for code-mixed queries. The Hindi components in both the tweets and the queries are written in Roman transliterated form. This subtask did not consider cases where both Roman and Devanagari scripts are present. Therefore, the documents in this case are tweets consisting of code-mixed Hindi-English texts where the Hindi terms are in Roman transliterated form. Given a query consisting of Hindi and English terms written in Roman script, the system has to retrieve the top-k documents (i.e., tweets) from a corpus that contains Code-Mixed HindiEnglish tweets. The expected output is a ranked list of the top twenty (k=20 here) tweets retrieved from the given corpus. 3.1

Datasets

Initially we released 6,133 code-mixed Hindi-English tweets with 23 queries as the training dataset. Later we released a document collection containing 2,796 code-mixed tweets along with with 12 code-mixed queries as the testset. Query terms are mostly named entities with Roman transliterated Hindi words. The average length of the queries in the training set is 3.43 words and in the testset it is 3.25 words. The tweets in the training set cover 10 topics whereas the testset cover 3 topics. 3.2

Submissions

This year total 7 teams have submitted 13 runs. The submitted runs for the retrieval task of Code-Mixed tweets mostly adopted preprocessing of the data and then applying di erent techniques for retrieving the desired tweets. Team Amrita CEN [ 4 ] removed some Hindi/English stop words to declutter useless words. After that, they have tokenized all the tweets. The cosine distance was used to score the relevance of tweets to the query. After that, the top 20 tweets based on the scores were retrieved. Team CEN@Amrita[14] used a Vector Space Model based approach. Team UB [ 11 ] has adopted three di erent techniques for the retrieval task. First, they have used Named Entity boosts where the purpose was to boost the documents based on their NE matches from the query, i.e., the query was parsed to extract NEs and each document (tweet) that matched the given NE was provided a small numeric boost. At the second level of boosting, phrase matching was done , i.e., documents that more closely matched the input query phrase were ranked higher than those that did not. The UB team used Synonym Expansion and Narrative based weighting as the second and third techniques. Team NITA NITMZ performed stop words removal followed by query segmentation and nally merging. Team IIT(ISM) D considered every tweet as a document and indexed using uniword indexing on Terrier implementation. Query terms were expanded using soundex coding scheme. Terms with identical soundex code were selected as candidate query and included in nal queries to retrieve the relevent tweets (documents). Further, they have used three di erent retrieval models BM25, DFR and TF-IDF to measure the similarity. However, this team submitted the runs after the deadline. 3.3

Results

The retrieval task requires that the retrieved documents at higher ranks be more important than the retrieved documents at lower ranks for a given query and we want our measures to account for that. Therefore, set based evaluation metrics such as Precision, Recall and F-measure are not suitable for this task. Therefore, we used Mean Average Precision (MAP) as the performance evaluation metric for subtask-2. MAP is also referred to as \average precision at seen relevant documents". The idea is that, rst, average precision is computed for each query and subsequently the average precisions are averaged over the queries. MAP is represented as

M AP =

N j=1 Qj i=1

N Qj 1 X 1 X P (doci) where Qj refers to the number of relevant documents for query j, N indicates the number of queries and P (doci) represents precision at the ith relevant document.

The evaluation results of the submitted runs are reported in Table 5. The highest MAP (0.0377) was achieved by team Amrita@CEN which is still very low. The signi cantly low MAP values in Table 5 suggest that the task of retrieving Code-Mixed tweets against query terms comprising codemixed Hindi and English words is a di cult task and the techniques proposed by the teams do not produce satisfactory results. Therefore, the problem of retrieving relevant code-mixed tweets requires better techniques and methodologies to be developed for improving system performance.

4. SUMMARY

In this overview, we elaborated on the two subtasks of the MSIR-2016 at FIRE-2016. The overview is divided into two major parts one for each subtask, where the dataset, evaluations metric and results are discussed in detail. A total of 33 runs were submitted from 9 unique teams.

In subtask-1, 20 runs were received from 7 teams. The best performing team achieved 83.333% accuracy. The average question classi cation performance obtained in terms of accuracy was 78.19% which was quite satisfactory considering this new research problem. The subtask-1 deals with code-mixed Bengali-English language. In the coming years, we would like to include more Indian languages. The participation was encouraging and we plan to continue the subtask-1 in subsequent FIRE conferences.

Subtask-2 received a total of 13 run submissions from 7 teams out of which one team submitted after the deadline. The best MAP value achieved was 0.0377 which is considerably low. From the results of the run submissions it can be inferred that information retrieval of code-mixed informal micro blog texts such as tweets is a very challenging task. Therefore, the stated problem opens and calls for new avenues of research for developing better techniques and methodologies.

ACKNOWLEDGMENTS

Somnath Banerjee, Sudip Kumar Naskar and Sivaji Bandyopadhyay acknowledge the support of the Ministry of Electronics and Information Technology (MeitY), Government of India, through the project \CLIA System Phase II".

The work of Paolo Rosso has been partially funded by SomEMBED MINECO TIN2015-71147-C2-1-P research project and by the Generalitat Valenciana under the grant ALMAMATER (PrometeoII/2014/030).

We would also like to thank everybody who helped spread awareness about this track in their respective capacities and the entire FIRE team for giving us the opportunity and platform for conducting this new track smoothly. 6. Classi cation. In Working notes of FIRE 2016 Forum for Information Retrieval Evaluation, Kolkata, India, December 7-10, 2016, CEUR Workshop

Proceedings. CEUR-WS.org, 2016. [13] R. S. Roy, M. Choudhury, P. Majumder, and

K. Agarwal. Overview and datasets of FIRE 2013 track on transliterated search. In Fifth Forum for

Information Retrieval Evaluation, 2013. [14] S. Singh and Anand Kumar, M and Soman, KP.

CEN@Amrita: Information Retrieval on CodeMixed Hindi-English Tweets Using Vector Space Models. In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation, Kolkata, India, December 7-10, 2016, CEUR Workshop Proceedings, December 2016. [15] A. Saini. Code Mixed Cross Script Question

Classi cation. In Working notes of FIRE 2016 Forum for Information Retrieval Evaluation, Kolkata, India, December 7-10, 2016, CEUR Workshop

Proceedings. CEUR-WS.org, 2016. [16] R. Sequiera, M. Choudhury, P. Gupta, P. Rosso, S. Kumar, S. Banerjee, S. K. Naskar, S. Bandyopadhyay, G. Chittaranjan, A. Das, and K. Chakma. Overview of FIRE-2015 Shared Task on Mixed Script Information Retrieval.

References

[1] U. Z. Ahmed , K. Bali , M. Choudhury , and S. VB. Challenges in designing input method editors for indian languages: The role of word-origin and context . Advances in Text Input Methods (WTIM 2011 ), pages 1 {9 , 2011 .

[2] M. Anand Kumar and K. P. Soman. Amrita-CEN@ MSIR-FIRE2016: Code-Mixed Question Classi cation using BoWs and RNN Embeddings . In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation , Kolkata, India, December 7- 10 , 2016 , CEUR Workshop Proceedings . CEUR-WS.org, 2016 .

[3] S. Banerjee , S. K. Naskar , P. Rosso , and S. Bandyopadhyay. The First Cross-Script Code-Mixed Question Answering Corpus. Proceedings of the workshop on Modeling, Learning and Mining for Cross/Multilinguality (MultiLingMine 2016 ), co-located with The 38th European Conference on Information Retrieval (ECIR) , 2016 .

[4] H. B. Barathi Ganesh , M. Anand Kumar , and K. P. Soman . Distributional Semantic Representation for Text Classi cation and Information Retrieval . In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation , Kolkata, India, December 7- 10 , 2016 , CEUR Workshop Proceedings . CEUR-WS.org, 2016 .

[5] R. Bhargava , S. Khandelwal , A. Bhatia , and Y. Sharmai . Modeling Classi er for Code Mixed Cross Script Questions . In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation , Kolkata, India, December 7- 10 , 2016 , CEUR Workshop Proceedings . CEUR-WS.org, 2016 .

[6] D. Bhattacharjee and P. Bhattacharya . Ensemble Classi er based approach for Code-Mixed Cross-Script Question Classi cation . In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation , Kolkata, India, December 7- 10 , 2016 , CEUR Workshop Proceedings . CEUR-WS.org, 2016 .

[7] K. Chakma and A. Das. CMIR: A Corpus for Evaluation of Code Mixed Information Retrieval of Hindi-English Tweets . In In the 17th International Conference on Intelligent Text Processing and Computational Linguistics (CICLING) , April 2016 .

[8] M. Choudhury , G. C. Gupta , P. Gupta , and A. Das . Overview of FIRE 2014 Track on Transliterated Search , 2014 .

[9] P. Gupta , K. Bali , R. E. Banchs , M. Choudhury , and P. Rosso . Query expansion for mixed-script information retrieval . In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , pages 677 { 686 . ACM, 2014 .

[10] X. Li and D. Roth . Learning question classi ers . In Proceedings of the 19th international conference on Computational linguistics-Volume 1 , pages 1 { 7. Association for Computational Linguistics, 2002 .

[11] N. Londhe and R. K. Srihari. Exploiting Named Entity Mentions Towards Code Mixed IR : Working Notes for the UB system submission for MSIR@FIRE'16.

[12] G. Majumder and P. Pakray. NLP-NITMZ @ MSIR 2016 System for Code-Mixed Cross-Script Question