This paper describes our experiment on two cross-lingual and one monolingual English text retrievals at CLEF1 in the ad-hoc track. The cross-language task includes the retrieval of English documents in response to queries in two most widely spoken Indian languages, Hindi and Bengali. For our experiment, we had access to a HindiEnglish bilingual lexicon, 'Shabdanjali', consisting of approx. 26K Hindi words. But neither we had any effective Bengali-English bilingual lexicon nor any parallel corpora to build the statistical lexicon. Under this limited resources, we mostly depended on our phoneme-based transliterations to generate equivalent English query from Hindi and Bengali topics. We adopted Automatic Query Generation and Machine Translation approach for our experiment. Other language-specific resources included a Bengali morphological analyzer, a Hindi stemmer and a set of 200 Hindi and 273 Bengali stopwords. Lucene framework was used for stemming, indexing, retrieval and scoring of the corpus documents. The CLEF results suggested the need for a rich bilingual lexicon for CLIR involving Indian languages. The best MAP values for Bengali, Hindi and English queries for our experiment were 7.26, 4.77 and 36.49 respectively.
Cross-language (or cross-lingual) Information Retrieval (CLIR) involves the study of retrieving the documents in a language other than the query language. Since the language of query and documents to be retrieved are different, either the documents or queries need to be translated in CLIR. But this translation step tends to cause a reduction in the retrieval performance of
CLIR as compared to monolingual information retrieval. A study in [
Oard [
This paper presents two cross-lingual and one English monolingual text retrieval. The crosslanguage task includes English document retrieval in response to queries in two Indian languages: Hindi and Bengali. Although Hindi is mostly spoken in north India and Bengali in the Eastern India and Bangladesh only, the former is the fifth most widely spoken language in the world and Bengali the seventh. This requires attention on CLIR involving these languages. In this paper, we restrict ourselves to Cross-Language text retrieval applying Machine Translation approach.
The rest of the paper is structured as follows. Section 2 briefly presents some of the works on CLIR involving Indian languages. The next section provides the language specific and open source resources used for our experiment. Section 4 builds our CLIR model on the resources and explains our approach. CLEF evaluations of our results and their discussions are presented in the subsequent section. We conclude this paper with a set of inferences and scope of future works. 2
Cross-language retrieval is a budding field in India and the works are still in its primitive state.
The first major work involving Hindi occurred during TIDES Surprise Language exercise in a
one month period. The objective of the exercise was to retrieve Hindi documents, provided by
LDC (Linguistic Data Consortium), in response to English queries. The participants used parallel
corpora based approach to build the statistical lexicon [
We used various language specific resources and open source tools for our Cross Language
Infomation Retrieval (CLIR) experiments. For the processing of English query and corpus, we used
the stop-word list (33 words) and porter stemmer of Lucene framework. For Bengali query, a
Bengali-English transliteration (ITRANS) tool2 [
3The list was provided by Jadavpur University, Kolkata. an open source Bengali-English bio-chemical lexicon ( 9k Bengali words) and a Bengali morphological analyzer of moderate performance were used. Hindi language specific resources included a Hindi-English Transliteration tool (wx and ITRANS), a Hindi stop-word list of 200 words, a Hindi-English bilingual lexicon ’Shabdanjali’ containing approximately 26K Hindi words and a Hindi Stemmer4. We also manually built a named entity list of 1510 entries mainly drawn from the names of countries and cities, abbreviations, companies, medical terms, rivers, seven wonders, global awards, tourist spots, diseases, events of 2002 from wiki etc. Finally, the open source Lucene framework was used for indexing and retrieval of the documents with their corresponding scores. 4
The objective of Ad-Hoc Bilingual (X2EN) and Monolingual English tasks was to retrieve the relevant documents from English target collection and submit the results in ranked order. The topic sets for these two tasks consist of 50 topics and the participant is asked to retrieve at least 1000 documents from the corpus per query for each of the source languages. Each topic consists of three fields: a brief ’title’, almost equivalent to a query provided by the end-user to a search engine; a one-sentence ’description’, specifying more accurately what kind of documents the user is looking for from the search and a ’narrative’ for relevance judgements, describing what is relevant to the the topic and what is not. Our approach to the problem can be broken into 3 phases: corpus processing, query generation and document retrieval. 4.1
The English news corpus of LA Times 2002, provided by CLEF, contained 1,35,153 documents of 433.5 MB size. After removing stop words and stemming the documents, they were indexed using the Lucene indexer to obtain the index terms corresponding to the documents. 4.2
We adopted Automatic Query Generation method to immitate the possible application of CLIR
on the web. The language-specific stop-words were first removed from the topics. To remove the
most frequent suffixes, we used a morphological analyzer for Bengali, a stemmer for Hindi and
the Lucene stemmer for English topics. We considered all possible stems for a single term as no
training data was available to pick the most relevant stem. This constitutes the final query for
English monolingual run. For Indian languages, the stemmed terms were then looked up in the
bilingual lexicon for their translations into English. All the translations for the term were used for
the query generation (Structured Query Translation), if the term was found in the lexicon. But
many terms did not occur in the lexicon due to its limitation in size or the improper stemming
or as the term is a named entity [
Note that we did not expand the query using the Pseudo Relevance Feedback (PRF). This
is due to the fact that it does not improve the retrieval significantly for CLIR, rather hurts by
increasing noise [
The query generated in the above phase is fed into Lucene search engine and the documents were retrieved along with their normalized scores. Lucene scorer follows the Vector Space Model (VSM) of Information Retrieval.
The evaluation document set for the ad-hoc bilingual and monolingual tracks consists of 1,35,153
documents from LA Times 2002. For the 50 topics originally provided by CLEF, there were
manually selected 2247 relevant documents which were matched against the retrieved documents
of the participants. We provided the set of 50 queries to the system for each run of our experiments.
Six official runs were submitted for the Indian langauges to English bilingual retrieval, three for
Hindi queries and three for Bengali queries. Three monolingual English runs were also submitted
to compare the results between bilingual and monolingul retrievals. The runs were performed
using only <title> field, <title+desc> fields and <title+desc+narr> fields per topic for each of
these languages. The performance metrics for the nine runs of our experiments are presented in
the following tables.
<title>
<title+desc>
Table 1 presents four basic primary metrics for CLIR, viz., MAP (Mean Average Precision),
GMAP (Geometric Mean Average Precision), B-Preference and Precision at 10 retrieved
documents (P@10) for all of our official runs. The lower values of the GMAP corresponding to MAP
clearly specifies the poor performance of our retrievals in the lower end of the average precision
scale. Also, lower values of the MAP for Hindi than the work of [
Table 2 shows the results of the topicwise score breakup for the relevant 2247 documents. As seen from the table, number of failed topics (with no relevant retrieval) and topics with MAP ≤ 10% gradually decreased with more fields from the topic, thus establishing the fact again mentioned in the earlier paragraph. Also, the better result for Hindi than Bengali is clearly attributed to its better lexicon. But when it comes to the number of topics with MAP ≥ 50%, Bengali clearly outperforms Hindi due to the n´oise factor´, mentioned in the previous paragraph. A careful analysis of the queries revealed that the queries with named entities provided better results for all the runs, whereas the queries without named entities performed very poor due to poor bilingual lexicons and thus brininging down the overall performance metrics. This clearly implies the importance of a very good bilingual lexicon and transliteration tool in the CLIR for Indian languages. Recall is a very important performance metric for CLIR specifically for the case when the number of relevant documents is significantly low compared to the target collection (in this case, it is 1.66% only). It is noteworthy that the recall value has improved even for the <title+desc+narr> field compared to other runs and Bengali has again outperformed Hindi due to n´oise factor´.
The recall vs average precision graphs in Figure 1 and retrieved documents vs precision graphs in Figure 2 suggest the need for refinement of important query terms (e.g. named entity) and weigh them more than the translated terms. Also, we used Structured Query Translation and assigned uniform weight on all of them. But this has affected the precision values for some queries even when the recall is significantly high. Again, we used all possible stems for a term when multiple stems are possible and this has also added to the lower precision values for some queries. A proper named entity recognizer is also important to prune out the named entities from other non-lexical terms. All of these will decrease the noise in the final query and thereby help the lucene ranking algorithm to push the relevant documents to the top.
This was our first participation in CLEF and we performed our experiment under a limited resource
scenario with a very basic Machine Translation approach. But the experiment pointed out the
necessity of good language-specific resources, specifically a rich bilingual lexicon. A close analysis
between cross-lingual and monolingual retrievals clarly pointed out the importance of four factors
in CLIR, mentiond earlier [