This paper describes our ¯rst participation in the Indian language sub-task of the main Adhoc monolingual and bilingual track in CLEF1 competition. In this track, the task is to retrieve relevant documents from an English corpus in response to a query expressed in di®erent Indian languages including Hindi, Tamil, Telugu, Bengali and Marathi. Groups participating in this track are required to submit a English to English monolingual run and a Hindi to English bilingual run with optional runs in rest of the languages. We had submitted a monolingual English run and a Hindi to English cross-lingual run. We used a word alignment table that was learnt by a Statistical Machine Translation (SMT) system trained on aligned parallel sentences, to map a query in source language into an equivalent query in the language of the target document collection. The relevant documents are then retrieved using a Language Modeling based retrieval algorithm. On CLEF 2007 data set, our o±cial cross-lingual performance was 54.4% of the monolingual performance and in the post submission experiments we found that it can be signi¯cantly improved up to 73.4%.
The rapidly changing demographics of the internet population [
Lack of resources is still a major reason for relatively less number of e®orts in the cross-lingual
setting in Indian subcontinent. Research communities working in Indian Languages, especially on
Machine Translation (MT) [
The rest of the paper is organized as follows. We will ¯rst de¯ne the problem in section 2, followed by the presentation of our approach in section 3. In sections 4.1 & 4.2 we will describe data set along with the resources used and present the performance of our system (section 4.3) in the CLEF competition. Section 4 includes some analysis of the results. Section 5 presents our conclusion and identi¯es our plans for future work. 2
Cross Language Evaluation Forum (CLEF) aims at promoting research in the design of multilingual, multi-modal retrieval systems by providing an opportunity for the research communities working in di®erent languages to collaborate and share their experiences. Each year it organizes a series of evaluation tracks to test di®erent aspects of cross-language information retrieval system development.
We have participated in the CLEF competition, speci¯cally in the Indian Language sub-task of the main CLEF 2007 Ad-hoc monolingual and bilingual track. This track tests the performance of systems in retrieving the relevant documents in response to a query in the same and di®erent language from that of the document set. In the Indian language track, documents are provided in English and queries are speci¯ed in di®erent languages including Hindi, Telugu, Bengali, Marathi and Tamil. The system has to retrieve 1000 relevant documents as response to a query in any of the above mentioned languages. All the systems participating in this track are required to submit a English to English monolingual run and a Hindi to English bilingual run. Runs in rest of the languages are optional. 3
Converting the information expressed in di®erent languages to a common representation is
inherent to cross-lingual applications to build the language barrier. In CLIR, either the query or the
document or both need to be mapped into the common representation to retrieve the relevant
documents. Translating all documents into the query language is less desirable due to the enormous
resource requirements. Usually the query is translated into the language of the target collection
of documents. Typically three types of resources are exploited for translating the queries:
bilingual machine readable dictionaries, parallel texts and machine translation systems. MT systems
typically produce one candidate translation thus some potential information which could be of
use to IR system is lost. Researchers [
In our work, we have used a statistically aligned Hindi to English word alignments that were
learnt during the training phase of machine translation. The query in Hindi language is translated
into English using word by word translation. For a given Hindi word, all English words which
have translation probability above certain threshold are selected as candidate translations. Only
top `n' of these candidates are selected as ¯nal translations to reduce ambiguity in the translation.
The aligned bilingual dictionary may not contain some of the query words because either the
word is not available in the parallel corpus or the translation probabilities are less than the
threshold. In such cases, we attempt to transliterate the query word into English. We have used a
noisy channel model based transliteration algorithm [
Once the query is translated into the language of the document collection, standard IR
algorithms can be used to retrieve the relevant documents. We have used Langauge Modeling [
In a nutshell, structural query translation [
In both the Adhoc bilingual `X' to English track and Indian language sub track, the target
document collection consisted of 135,153 English news articles published in Los Angeles Times, from
the year 2002. During the indexing of this document collection, only text portion (embedded in
<LD> and <TE> tags) was considered. Note that the results reported in this paper does not
make use of other potentially useful information present in the document, such as, the document
heading (with in <DH> tag) and the photo caption (in <CP> tag), even though we believe that
including such an information would improve the performance of the system. The resulting 85,994
non-empty documents were then processed to remove the stop words and the remaining words
were reduced into their base form using Porter stemmer [
50 topics originally created in English and translated later into other languages were distributed among the participants. For processing Hindi queries, a list of stop words was formed based on the frequency of word in the monolingual corpus obtained corresponding to the Hindi part of the parallel data. This list was then used to remove any less informative words occurring in the topic statements. The processed query was then translated into English using the word alignment table. 4.2
We have used the word alignment table that was learnt by the SMT [
Each of the participating systems was required to submit 1000 relevant documents for each topic. For each query, a pool of candidate relevant documents is created by combining the documents submitted by all systems. From the pool of such candidate documents assessors ¯lter out actual relevant documents from non relevant ones. These relevance judgements are then used to automatically evaluate the quality of cross-lingual retrieval of participating CLIR systems.
In this section we discuss the results of our monolingual English run and Hindi to English bilingual run. In our case we speci¯cally participated in only one Indian language - Hindi, though the data was available in 5 Indian languages. For our o±cial submission, with the aim of reducing noise in the translated query, we have used a relatively high threshold of 0.3 for the translation probability. To avoid ambiguity, when there are many possible English translations for a given Hindi word, we allowed only 2 best possible translations according to the statistical alignments learnt by SMT. Table 2 shows the o±cial results of our submission. We have submitted di®erent runs using title, description (td) and title, description and narration (tdn) as query. Narration seems to be improving the cross-lingual retrieval performance, in terms of Mean Average Precision (MAP), more than that in monolingual setting. In the rest of the experiments it is assumed that narration is included as a part of the query unless explicitly mentioned. Figure 1 shows a comparison of cross-lingual and mono-lingual submissions in terms of precision at di®erent levels of interpolated recall.
In a second set of experiments, we experimented with various levels of threshold and the number of translations above the threshold and their e®ect on MAP score. The results obtained by monolingual system and cross-lingual system with varying threshold are shown plotted in Figure 2. The x-axis represents the number of top words considered when many of the target language words have translation probabilities above the chosen threshold. Note that y-axis represents only a subset (0.15-0.45) of the entire possible range (0-1) in which a MAP score can lie. The right most bar represents the monolingual performance of the system. The ¯gure shows that the performance increases as the threshold decrease but again it drops if you consider all the possible translations (dip when 10 and 15 translations were considered) perhaps due to the shift in query focus with the inclusion of many less synonymous target language words. For the CLEF data set, we found that, considering 4 most possible translations with out any threshold (left most bar) on the translation probability gave us the best results (73.4% of monolingual IR performance).
As the threshold decrease potentially two things can happen; words which were not translated
previously can get translated or new target language words whose translation probability was
below the threshold will now become the candidates of the translated query. Table 3 shows the
fraction of query words that were translated corresponding to each of these thresholds. Table 3
and Figure 2 show that as the threshold on the translation probability decrease, fraction of query
words getting translated increases, resulting in an overall increase in system performance. But the
performance increase between having no threshold and a threshold of 0.1 compared to the small
fraction of new words that got translated suggest that even noisy translations, even though they
are not true synonymous, might help CLIR. This perhaps due to the fact that for the purposes of
IR, having a list of associated words may be su±cient to identify the context of the query [
This paper describes our ¯rst attempts in building a CLIR system with the help of a word alignment table learned, from a parallel corpora, primarily for statistical machine translation. We present our experience and results of our participation in the Indian language sub-task of the Adhoc monolingual and bilingual track of CLEF 2007. In post submission experiments we found that, on CLEF data set, a Hindi to English cross lingual information retrieval system using a simple word by word translation of the query with the help of a word alignment table, was able to achieve » 73% of the performance of the monolingual system. Empirically we found that considering 4 most probable word translations with no threshold on the translation probability gave the best results.
Since the quality of the dictionary will a®ect the performance of the system, in future we would like to explore the e®ect of size and quality of the parallel data on the word alignments, and subsequently on the CLIR performance. We would also like to compare the use of a statistically learned word alignments with respect to a hand crafted dictionary of similar size for CLIR application.