In this year's CLEF submissions we focus on using a state-of-the-art statistical machine translation approach for ad-hoc cross-language retrieval. Our machine translation approach is phrase-based as opposed to statistical word-based approaches that have been previously used for query translation in cross-language IR. The phrase translation probabilities were estimated by using the Europarl corpus. For query formulation, we also use the n-best lists of translation candidates to assign weights to query terms. Our results show that a statistical phrase-based approach is a competitive alternative to commercial, rule-based machine translation approaches in the context of cross-language IR.
3. Use of a parallel corpus to estimate the probabilities that word w in the source language translates into word w0 in the target language.
The main shortcoming of most commercial machine translation systems is that they only return the most likely translation, where ‘most likely’ is defined in terms of the internal algorithm of the translation system. But this does not mean that there are no other equally good—or, by some other objective standard, maybe even better—alternative translations.
On the other hand, there are several drawbacks to the parallel-corpus approach as well. First, although
parallel corpora are available for many of the European languages, there are many languages for which there
are still no parallel corpora large enough to estimate translation probabilities. Second, most of the parallel
corpora belong to a rather specific domain, such as the Europarl corpus,1 which contains the proceedings
of the European parliament in 11 languages for the years 1996–2003. This introduces a bias toward the
domain of the parallel corpus and makes the learned translation probabilities less reliable for other domains.
A third disadvantage is that the translation probabilities induced from parallel corpora are typically based
on single-word mappings, although recent template-based statistical methods facilitate the acquisition of
phrase translations [
Here, we propose an approach that uses a state-of-the-art phrase based statistical machine translation system. Since this system is trained on the aforementioned Europarl corpus, we wanted to investigate whether such a translation approach results in a well-performing cross-language IR system despite the difference in genre between the corpus that is used to train the translation model and the retrieval corpus.
This paper is organized as follows: The next section provides some background on statistical machine translation and its application to cross-language IR. Section 3 describes the experimental set-up and results of our runs submitted to CLEF 2006. Finally we draw some conclusions in Section 4. 2
This section provides some background on statistical machine translation and how it is applied to crosslanguage retrieval. 2.1
Word-based statistical machine translation approaches have been used in several cross-language IR
systems, see, e.g. [
In the area of machine translation word-based translation approaches more recently have been replaced
by phrase-bases approaches, see, e.g. [
Since we are only interested in finding the most likely English translation for a given foreign sentence, p( f ) can be disregarded: argmaxe p(e| f ) = argmaxe p( f |e) · p(e) (2) Here, the likelihood, p(e| f ), models the faithfulness of translation, i.e. to what extent it covers the semantic content of the foreign sentence, and the prior, p(e), models the fluency of the generated translation.
During decoding, which is the process of finding the English sentence maximizing p(e| f ) according
to equation (2), English translation candidates are generated incrementally by applying matching phrase
translations and considering the probability of the fluency of each translation candidate. A number of
decoding approaches are used in statistical machine translation, and here we follow the multi-stack beam
search design used in [
In multi-stack beam search decoding partially generated translation candidates are grouped in stacks,
where candidates are put into the same stack if they have translated the same number of words from the
foreign sentence. Since comprehensive search is computationally prohibitive, and in fact NP-complete, see
[
Although most alignment approaches, such as the IBM models 1–5 [
In this approach, first the intersection of the alignment links of both alignment directions, i.e. e-to- f and f -to- e, is taken. This initial alignment is then expanded by adding links that are adjacent to existing links, which is done iteratively until no further alignment links can be added.
During phrase extraction, groups of adjacent words are extracted on the foreign and the English side. Two phrases are considered translations of each other if the words in the phrase pair are only aligned to each other and not to words outside the phrase pair. The conditional phrase translation probabilities are simply computed by marginalizing the joint distribution of the phrase pair: p( f¯|e¯) =
freq( f¯, e¯)
∑ f¯ freq( f¯, e¯)
where f¯ is a phrase in the foreign language, and e¯ is an English phrase. For more details see [
When estimating the probability of a translation candidate, it is impossible to use maximum likelihood estimates on the entire string, by using the chain rule p(e) =
p(e1) · p(e2|e1) · p(e3|e1, e2) · . . . · p(en|e1, . . . , en−1) where e is a string in target language of length n, as one would run into serious data sparseness issues. Instead most language modeling approaches use a limited horizon of two words, called tri-gram models: p(e) ≈
p(e1) · p(e2|e1) · p(e3|e1, e2) · . . . · p(en|en−2, en−1)
Despite the limited context that is taken into account they still form a good compromise between prediction
quality, robustness and computational tractability.
(3)
(4)
2Pharaoh is freely available from http://www.isi.edu/licensed-sw/pharaoh/ .
During decoding the different probabilities are combined by using a log-linear model. In addition to the
translation and language model probabilities our model contains parameters that regulate, for example,
word re-ordering, length of the translation, and number of phrase matches. Manually estimating the
appropriate weights for all parameters is nearly impossible as the number of combinations is simply too large.
Hence some form of optimization is required. Here we use the minimum error rate training (MERT)
approach introduced by Franz Och [
When translating the queries, no stop word removal is applied. The only pre-processing that is applied is tokenization and case folding, as the entries in the phrase table are case-folded as well. The main reason for keeping stop words is that we want to keep the input to the MT system fluent.
One of the major disadvantages of using commercial machine translation systems is that they tend to return only the most likely translation, despite the fact that there are many more translations that might be almost as good and formulate the translation slightly different. In addition, if the machine translation system is mainly rule-based, scoring different translation candidates can be rather difficult as it is hard to consistently assign weights to human-generated translation rules.
On the other hand it is very easy to generate n-best list for statistical machine translation systems, as all translation and language model probabilities are estimated by using training data. 3
This year, we participated in the English-to-French and English-to-Portuguese cross-language retrieval tasks. The following three subsections describe the configurations and data used for our machine translation component and the information retrieval system, and report the official CLEF 2006 results for those runs. 3.1
For both translation directions we used the Europarl parallel corpus to estimate the phrase translation probabilities. Table 1 summarizes some collection statistics of the data used to build the machine translation systems for our cross-language retrieval experiments.
no. of sentence pairs no. of English words no. of foreign words no. of distinct phrase pairs no. of distinct phrase pairs (CLEF 2006) no. of words in language model corpus no. of distinct 1-grams no. of distinct 2-grams no. of distinct 3-grams
English-French 645,518 14.4M 16.3M 28.0M 0.6M 87.5M 512.9K 5.8M 5.8M
English-Portuguese 568,446 13.2M 13.5M 29.3M 0.6M 126.4M 509.7K 10.3M 10.3M 3We are grateful to Philipp Koehn who provided us with a re-implementation of Och’s optimization procedure.
The first five rows in Table 1 refer to parallel corpora from Europarl that were used to estimate phrase translation probabilities. The resulting phrase translation table is rather big and hard to keep in memory, and it was filtered to retain only phrases that match word sequences occurring in the CLEF 2006 topic set (see row 5).
In order to build the French and the Portuguese language models, we used the French and Portuguese
side of the parallel corpus in combination with the documents in the CLEF retrieval corpus: ATS 1994–
1995 and Le Monde 1994–1995 for French, and Fohla 1994–1995 and Publico 1994–1995 for Portuguese,
see Table 1 for the respective sizes. We used the SRI Language Modeling toolkit [
The parallel corpora were tokenized and case-folded. We used the simple rule-based tokenization scripts for all three languages involved. Diacritics were not removed.
For query translation we used the title and description fields only, and each field was translated separately. Note that in Section 2 we assumed that we translate from a foreign language into English, as most MT approaches focus on translation into English. For our experiments we translate from English into French and Portuguese, respectively. Hence English is referred to as the foreign language in the discussion below.
Parameter estimation for the translation models used CLEF test sets from previous years. Most of the CLEF topics are available in several languages where topics have been created by a human translator. For English-to-French translation we used the topics 210 from CLEF 2001–2004, and for English-toPortuguese translation we used the 100 topics from 2002 and 2004. 3.2
The submitted runs used FLEXIR, a home-grown information retrieval system [
The text collection files were pre-processed using the same French and Portuguese tokenizers that were used to pre-process the parallel corpus. Stop words were removed from the translated queries, using the stop word list provided by the University of Neuchatel.4
Stemming and n-gram splitting was used for the French run, where we used the S NOWBALL stemmer, which was provided by the University of Neuchatel as well. After stemming, n-gram splitting was applied with n = 4. Note that n-gram splitting, was only applied to individual tokens and did not cross word boundaries.
The Portuguese run used only ngram splitting and no stemming. Here n = 5 and n-grams did cross word boundaries. For both French and Portuguese, these settings were determined by using the CLEF 2005 test collection as a development set.
Blind relevance feedback was applied to expand the original query with related terms. Term weights
were computed by using the standard Rocchio method [
As we mentioned above, both topic fields, i.e., the title and the description, were translated separately. For retrieval both translations were merged into a single query. For our official submissions we also used the n-best lists generated by our statistical machine translation system. The n-best lists are used to assign weights to terms in the query: weight f (t) =
n ∑i=1 occurs in(t, si) · p(si| f )
n ∑i=1 ∑ t∈si p(si| f ) (5) 4The stop word lists are available at http://www.unine.ch/info/clef/. where si is the i-th translation candidate for f and f is the original foreign language topic (note: here, English is the foreign language as we translate from English into French and Portuguese). We experimented with n-best lists of different length, and setting n = 10 yielded the best results for the CLEF 2005 test set. The term weight in (5) is computed for each occurrence of a term in a query. We submitted two official runs to CLEF 2006, one for the English-to-French cross-lingual task and one for the English-to-Portuguese cross-lingual task. The MAP scores for both runs are shown in Table 2. Although using n-best machine translation output for term re-weighting did lead to slight improvements for the CLEF 2005 development set, the improvements were still rather small. In part this could be due to the small variation in the n-best lists. The differences between the translation candidates tend to be minute and are often causes by different derivations of almost identical translations. For example, Table 3 shows the top ten translations for topic 301.
Rank
Translation English: what brands are marketed by nestle around the world ? les marques sont commercialisees par nestle dans le monde ? trans=ce que sont marques commercialisees par nestle dans le monde ? les marques sont commercialisees par nestle dans le monde - ? ce que marques sont commercialisees par nestle dans le monde ? ce que sont marques commercialisees par nestle dans le monde - ? ce que marques sont commercialisees par nestle dans le monde - ? les marques sont commercialisees par nestle dans le monde ? ce qu ’ marques sont commercialisees par nestle dans le monde ? ce qui marques sont commercialisees par nestle dans le monde ? les marques sont commercialisees par nestle dans le monde entier ?
As one can see, the translation candidates are almost identical to each other, and there is little variation in terms of lexical translation choices. Using longer n-best lists did not change this fact. It seems that more elaborate selection strategies are needed to filter out translation candidates that are merely due to word-order or phrase-matching differences. 4
With our submissions we tried investigate to what extent statistical machine translation is effective in the context of cross-language information retrieval. In particular with respect to machine translation approaches that translate the entire topic as opposed to word-translation models like the simpler IBM models. One of the main advantages of statistical machine translation is the ease with which n-best translation output can be generated, but unfortunately we were only able to get minor improvements from this. More elaborate ways of exploiting n-best lists remains a topic that we want to pursue further in future research. This work has been supported, in part, by the Nuffield Foundation, Grant No. NAL/32720.