This paper describes the approach we have used for identification of languages for a set of terms written in Roman script and approaches for the retrieval in mixed script domain, in FIRE-2015. The first approach identifies the class (native language of terms and whether a term is any named entity or of any other type) of given terms/words. MaxEnt a supervised classifier has been used for the classification which performed best for strict f-measure NE has score is 0.46 and strict f-measure NE_P has score 0.24. For the MSIR subtask Divergence from Randomness (DFR) based approach is used and performed better with block indexing and query formulation. Overall scores of our submission on NDCG@10 0.4335, 0.5328, 0.4489 and 0.5369 for ISMD1, ISMD2, ISMD3 and ISMD4 respectively. .
With the development of the Web 2.0, user’s count on Social sites are increasingly becoming higher. They write messages (specially blogs and post) on sites (such as Twitter and Facebook) in their own languages preferably using Roman scripts (transformed form). These post might consist terms of Non-English (or terms from user’s native ) languages, a simple English word, a mixed language term (like gr8, 2moro) or a Named Entity (NE). Identification of such categories play significant role in Natural Language Processing (NLP). It doesn’t remains limited to the NLP but also used in other sub-domains of linguistic processing and Information Retrieval (IR).
Since, blog posts contain some important information that opens up the scope of IR in informal texts (in form of posts or massages). Raw blogs data often have some erroneous text. Hence, before applying any IR steps data must be preprocessed using some linguistic processing approaches.
There are huge collection of data on/of the Web for various information needs but the track for adhoc retrieval. For the retrieval, collection has documents written in two scripts: Roman (transliterated form of Hindi terms in Roman script) and Devanagari. In whole corpus, some document has information in Devanagari, some others has in Roman and rest of the document has information in mixed (transliterated and native ) scrip one after another. To maximize the number of most relevant documents on the Web (in web retrieval) or from the corpus (in ad-hoc retrieval) it is necessary to retrieve the documents of other language and/or script. It is important to discuss three terms monolingual, multilingual and mixed script retrieval. In IR monolingual means query and documents to be retrieved are in single language where as multilingual query and documents may be in written diferent language. But, the mixed script retrieval is slightly diferent than monolingual retrieval. In mixed script retrieval, system should retrieve the relevant documents of same language written in more than one script.
In FIRE-2015, for the Mixed Script Information Retrieval track participant has to design the system for term classiifcation and for the retrieval of relevant documents written in Devanagari script and in Roman script.
We have used query expansion to reformulate the seed (information need) for addressing the mixed script retrieval issues.
Further in Section 2, we discussed the task descriptions. Section 3 shows related work on and Section 4, describes our approaches for annotation and MSIR. In Section 5, we have discussed results and analyzed errors. Section 6, conclude the strategies with the direction of future work. 2.
The track, Shared Task on Mixed Script Information Retrieval (MSIR) has three subtasks: Query Word Labeling, Mixed Script Ad-hoc Retrieval and Mixed-script Question Answering. We have participated in first two subtask.
Query Word Labeling
Input:- Let Q be the query set containing n query word wi(1 i n) written in Roman script. The word wi 2 Q (w1; w2; : : : ; wn), could be standard English (en) words or transliterated from another language L = {Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (ka), Malayalam (ml), Marathi (mr), Tamil (ta), Telugu (te)} and some Named Entities (NE). The task is to label the words as En or a member of L depending on whether it an English word, or a transliterated L-language word. Input and expected result for an utterance is given below as an example.
Input: <utterance id=“1”> en en en en en X X en en en en en en NE en en X en </utterance>
Mixed Script Ad-hoc Retrieval
There are more than 66K documents and 25 queries (seeds). Documents are written in Devanagari script, Roman script or in mixed script. Here mixed script means a document has same content in two scripts one after another. Out of 25 queries seven are in Devanagari and others are in Roman script.
The goal of the task is for a given query system should produce set of relevant documents in ordered where on the most relevant document should should come at first position.
Subtask-1 accomplished in two phases: Word Labeling and transliteration of H labeled word to its native (Devanagari) script. 3.1
The labeling is concerned with the classification of a given word written in Roman script. Query words wi can be classified and annotated with corresponding classes manually or using machine learning based classifiers. Various classifiers are there for classification such as Support Vector Machine(SVM), Bayesian networks, Decision Trees, NaiveBayes, MaxEnt and Neural Networks.
King and Abney started for labeling the languages of
words in cross-lingual documents[
This subtask was introduced in FIRE-2013 [
Gupta et al. in 2014, approached MSIR using 2-gram
tf-idf and deep learning based query expansion [
Our approaches for the solution of Subtask-1 and Subtask2 have been described in subsections below. 4.1
We have considered word labeling as classification issue
for the tags annotation to the given terms wi. Terms can
be classified either manually or using any classifier. Manual
classification and tagging is not feasible on the large dataset.
MaxEnt a supervised classifier is used for classification and
labeling of words from utterances. The Stanford’s MaxEnt
implementation is used for this purpose [
For the classification, model was trained on development data and then terms from utterances of test dataset were classified based on extracted features during training. 4.1.1
For the training purpose input terms and annotations are tokenized and made align with proper tags.
Features value with default parameter were used some of which are listed below: useNGrams accept boolean value true or false to make features from letter n-grams where true is assigned here. usePrefixSufixNGrams makes features from prefix and sufix substrings of the string and accept boolean value where we have assigned true. maxNGramLeng takes integer value and size beyond the assigned number will not be used in the model. Maximum length 4-grams was used. minNGramLeng also takes integer number and n-grams below this size will not be used in the model. It must be a positive integer and we have set it 1. sigma is a parameter to several of the smoothing methods,usually gives a degree of smoothing as standard deviation. Here this number is 3.0. useQN accepts boolean value where true indicates QuasiNewton optimization will be used if it is set to true. tolerance is convergence tolerance in parameter optimization and set 1e-4.
Classification model was train on above parameter values and 23 classes were identified during the training. 4.1.2
Given terms from utterances of test dataset were tokenized and parsed on trained model. Tokens of test data are classified and annotated with diferent tags such as for Hindi terms hi, English terms en, proper names (name of the person NE_P, location NE_L). 4.2
Subtask-2 has queries for Hindi song lyrics, astrological data and movies reviews related documents retrieval. Proposed approach consist three modules: documents indexing, query formulation and documents retrieval. 4.2.1
Simple bags-of-words approach may retrieve noisy documents for lyrics retrieval. Because in lyrics consecutive terms are important as change in position changes the context of a song. Hence, we have chosen block indexing with block-size 2 words in addition simple indexing. Two approaches simple indexing (bags-of-words) and block indexing (phrase retrieval) were used to index the collection with block size one word and two words respectively. 4.2.2
As documents in the corpus are in mixed script, seed value only can’t give good result for retrieval. Hence, the query must be reformulated to enhance the performance of the system. In query formulation, script of the query is identified and then transliteration is extracted using Google transliteration API. There are many terms for which API gives more than one transliteration for such term first one is chosen. For the submission of run ISMD2 and ISMD4 we have used formulated mixed script query as shown in Table 1.
Poisson model with Laplace after-efect and
normalization 2 of Divergence From Randomness (DFR) framework
has been used to measure the similarity score between
documents d and query Q [
∑ (qtf n w (t; d)) qtf n = n2Q
qtf
qtfmax
where w(t; d) is the weight of the document d for a query
term t and qtf n is the normalized frequency of term t in the
query. And qtf is the original frequency of term t in the
query, and qtfmax is the maximum qtf of all the composing
terms of the query for details see[
Our approaches have been evaluated on the provided test data for query word labeling and MSIR. In both the subtasks our approaches performed moderate. 5.1
MaxEnt based classifier worked moderate as depicted in table 2. In some of measure our approach performed well with scores 0.46 strict f-measure NE and 0.24 in strict fmeasure NE_P. For some metrics we performed moderate and in others poor as well. Some terms are misclassified e.g. Input utterance: <utterance id=“186”> ei path jodi na shesh hoy lyrics </utterance>
Annotated utterance: <utterance id=“186”> bn hi bn bn bn bn en </utterance>
The token ‘path’ in input utterance should have a Bengali term and has same meaning in Hindi and English also but misclassified in Hindi due to ambiguity as same term exist in Hindi. But ‘path’ seems to be ‘poth’ in Bengali due to regional accent. 5.2
Submitted four Runs for subtask-2, with combinations of simple indexing and original query, simple indexing and formulated query, block (size=2 words) indexing and original query and block (with size=2 words) indexing and formulated query. From the score in Table 3 we have observe that Run with block indexing and formulated queries better and the order in higher to lower performance on NDCG@10 is ISM 4 > ISM 2 > ISM 3 > ISM 1.
Overall the retrieval approaches performed moderate compare to other teams. Some challenges remains un-addressed in approaches: spelling variation in transliterated (Roman) text, combined term ( such as ‘kabhi-kabhi’ could be ‘kabhi’, ‘kabhi’, ‘tujo’ could be ‘tu’, ‘jo’) and translation (some document consist information in another language such as सूय रेखा कक राश could be translated into Line of Sun for Cancer) of query text. One more challenging issue is partial transliteration and translation. For example query number 69, query is “shani dashaa today for a 20 year old” in that ifrst two tokens are Hindi terms. Hence either Hindi terms will be translated to English or other terms need to be translated into Hindi and then transliterate into Roman text. 6.
Our work comprises two subtasks annotation and retrieval. We have used learning based classifier for word labeling. Label accuracy was moderate for submitted runs. We identiifed some terms were incorrectly labeled by the classifier. Perhaps this happened due an important reason i.e. term ambiguity where same term exist in more then one classes. For the MSIR, simple and block indexing both used separately during document indexing. In the query formulation transliterations are extracted using Google API. To measure the similarity score a DFR framework is used which performed moderate. Some query expansion approach can address MSIR retrieval issues. In future we are looking to address the unresolved issues mentioned above.
Metric MIXesAccuracy
NEsAccuracy
NEsCorrect strict f-measure NE strict f-measure NE_L strict f-measure NE_P strict f-measure X strict f-measure bn strict f-measure en strict f-measure gu strict f-measure hi strict f-measure kn strict f-measure ml strict f-measure mr strict f-measure ta strict f-measure te
TokensAccuracy UtterancesAccuracy Average F-measure Weighted F-Measure Team ISMD1 ISMD2 ISMD3 ISMD4
Block_Size 1 word 1 word 2 words 2 words