This report describes our test on using statistical translation models for bilingual IR tasks in CLEF-2001. These translation models have been trained on a set of parallel web pages automatically mined from the Web. Our goal is to compare the following approaches: -using the original parallel corpora or a cleaned corpora to train translation models; -using the raw translation probabilities to weigh query words or combine the probabilities with IDF; -using different cut-off probability values in the translation models (i.e. delete the translations lower than a threshold). Our results show that: -the models trained on the original parallel corpus work better than those on the cleaned corpora; -the combination of the probabilities with IDF is beneficial; -and it is better to cut-off the translation models at a certain value (0.01 in our case) than not cut them.
There have been several experiments on using statistical translation models for CLIR [Franz98,Nie99]. It has been shown that with a proper use of the translation models, we can obtain effectiveness comparable to that using a good machine translation system. In our previous studies we have successfully mined several large sets of parallel web pages, and used them to train translation models for the following language pairs: French-English, German-English, Italian-English and Chinese-English. The model training on these corpora has been the same as that using a manually prepared parallel corpus. However, there are several differences between these corpora and a manually prepared corpus. In particular, the corpora with parallel web pages contain much more noise (non-parallel pairs).
Therefore, some special means seem to be necessary to explore the best uses of the parallel web pages in the CLIR tasks. In this report, we describe our attempts to make better translation models from the parallel corpora, and to make better use of the trained translation models in CLEF bilingual IR tasks. The following problems will be investigated in our tests:
-the clean up of the parallel corpora in order to create neater corpora; -the cut-off of translation models in order to eliminate unreliable translations; -the use of two-directional query translation; -and the combination of translation models with bilingual dictionaries to increase the coverage.
In the following sections, let us first recall briefly the corpora and the training of translation models. Then we will report each of the above experiments.
We used the same sets of parallel web pages as the last year. The three corpora of French, Italian and German with English have been constituted automatically [Chen00]. Their sizes are as follows: The training of translation models is as follows: Given a set of parallel texts in two languages, it is first aligned into parallel sentences. The criteria used in sentence alignment are the position of the sentence in the text (parallel sentences have similar positions in two parallel texts), the length of the sentence (they are also similar in length), and so on [Gale93]. In [Simard92], it is proposed that cognates may be used as an additional criterion. Cognates refers to the words (e.g. proper names) or symbols (e.g. numbers) that are identical (or very similar in form) in two languages. If two sentences contain such cognates, it provides additional evidence that they are parallel. It has been shown that the approach using cognates performs better than the one without cognates. Before the training of models, each corpus is aligned into parallel sentences using cognate-based alignment algorithm.
Once a set of parallel sentences is obtained, word translation relations are estimated. First, it is assumed that every word in a sentence may be the translation of every word in its parallel sentence. Therefore, the more two words appear often in parallel sentences, the more they are thought of to be translation of one another. In this way, we obtain the initial probabilities of word translation.
At the second step, the probabilities are submitted to a process of Expectation Maximization (EM) in order to maximize the probabilities with respect to the given parallel sentences. The algorithm of EM is described in [Brown93]. The final result is a probability function P(f|e) which gives the probability that f is the translation of e. Using this function, we can determine a set of probable word translations in the target language for each source word, or for a complete query in the source language.
A translation model contains translations for every lexical item encountered in a training corpus, even if the item appeared only once. In the case where an item does not appear often, its translation is often sparse, i.e. it is translated by many different words with low probabilities. In fact, if a source word occurs only once in the training corpus, it may be translated by every word in the target sentence with quite comparable probabilities. Such translations are not reliable. In addition, storing the translations for every word requires a large space. It is not practical to use such a translation model in a real application. For example, the original French-English model contains more than 13 millions entries. Therefore, we exploited the possibility of reducing the translation models by cutting off the translations that we judge unreliable.
Two ways to cut off a model are investigated:
-We only keep a fixed number of lexical items in the model; -We remove all the translations below a certain threshold of translation probability. We tested several numbers in the first cut-off method: 1 million (1M), 100 thousands (100K), 10 thousands (10K), 5 thousands (5K) and 1 thousand (1K), and the thresholds tested are 0.05, 0.1 and 0.25. Below is a summary of the results with the CLEF-2000 data1 . We can see that the cut-off by probability is a better solution. In particular, when the threshold is set at 0.1, we obtain generally good results. In fact, the unreliable translations usually have low probabilities. So by eliminating low-probability translations, a large part of unreliable translations is removed.
Given that the words used in the CLEF queries are not rare words, we did not encounter the problem of increasing unknown words by cutting off the models.
The parallel corpora gathered from the Web contain a certain proportion of non-parallel texts. In fact, it is impossible to eliminate all the non-parallel texts in such a corpus automatically. The question we raised is whether it is helpful to use additional criteria to filter the corpora so that certain non-parallel texts can be eliminated. We did this filtering first on the Chinese-English corpus, also gathered with the same mining tool. The filtering criteria are the following [Nie01]:
-Tighten the control on text length: If the lengths of two presumably parallel texts are too different from the typical length ratio of the two language, then the texts are considered to be non-parallel. -Control on empty alignment: Once an automatic alignment algorithm is applied on a pair of texts, if the proportion of empty alignments (a sentence is aligned with nothing, or n:0 and 0:n alignments) is higher than a threshold, then the pair is eliminated. -Using a bilingual dictionary in alignment: In order to increase the accuracy of sentence alignment, we also examine the proportion of words in the sentence that have known translations (stored in the dictionary) in the corresponding sentence. This proportion is taken into account and used in combination with the length criterion in a length-based alignment algorithm [Gale93]. In our tests on Chinese-English corpus, we found that after the filtering, both the translation accuracy judged with 200 randomly selected words and the effectiveness of CLIR between English and Chinese have been increased significantly. Below are the best improvements we have been able to obtain: The results show a large drop of effectiveness in the de-en case. In the case of it-en, it is quite comparable to that without filtering. This counter-performance may be due to several factors:
-The parameters determined for the Chinese-English corpus (that are used during the filtering) are not necessarily suitable to the corpora of European languages; -There may be a flaw in the training of the translation models, as this was done in a hurry in the last minutes. We are now investigating both factors, and hope to find an explanation soon.
It is observed that certain common English words often appear as one of the top-raked translations of queries (e.g. make, provide, due, etc.). This is because these words frequently appear in the training corpora, and they are not considered as stopwords. So they are considered to be highly probable translations for many words in French, German and Italian. However, if we also consider translations of these English words back to the source languages, it would likely be translated to many different words, i.e. their translations would not be concentrated on the same source words for which they have been suggested as translations. Therefore, a combination of both directions in translation could eliminate such common words in query translations. This is the intuition of using the two-direction translation of queries.
In our implementation, the translation probabilities of both directions are multiplied, i.e. for a pair (e, f) of English and French, the probability of P(e|f) is multiplied by the probabilities of Σf' P(f'|e) where f' is all the words in the original French query.
However, our test did not show that this idea or this implementation works well. We observe a large decrease in performance compared with Tables 2 and 3. A possible reason is the filtering of the translations: after the translation of each query, we only keep the translations whose probability is higher than a certain threshold (0.001). So in many cases, there are much less than 50 translation words (as in the case of one-directional translations). As a matter of fact, the average numbers of English translation words per query are about 14 for French queries, less than 17 for German queries and less than 16 for Italian queries. This shows that the way we used to keep translation words is too selective. As While we observe that many false translation words have been eliminated, some new ones have been introduced (e.g. cad, part, and sedan), and the word "architettura" is translated by itself, probably due to the appearances of this word in English texts. So globally, we did not obtain any benefit from this two-directional translation. We will experiments some variances of this method in order to know if it is the idea or our particular implementation method that did not work well.
In the past, we have found that we can greatly improve the CLIR effectiveness by combining a translation model with a bilingual dictionary. The method we tested is to assign a fixed value to those translations that are found in the dictionary. In other words, if a translation is stored in the dictionary, then the weight of this translation will be incremented by a fixed value.
We observed that common words often have more translations than specialised words. As a result, common words will be represented by more translations, and indirectly, their importance in the query is increased unduly. In order to take into account the commonness of a translation word, we combine the translation probabilities with the idf value of the translation word. The intuition is to increase the probabilities of the uncommon translation words and lower the probabilities of common translation words.
We submitted 9 bilingual runs, three for each of the language pairs: fr-en, de-en and it-en.
The first group of runs uses only the 50 first translation words and the translation probabilities. The translation models used are cut-off by a threshold of 0.1 (P≥0.1). These runs are prefixed by RaliP01.
The second group combines the models used in the previous group with dictionaries -FreeDict that we found on the Web ( http://www.freedict.com/dictionary/index.html) . In particular, we used the dictionaries of French to English, German to English and Italian to English. All the translations in the dictionaries are assigned a fixed value (0.001). These runs are prefixed by RaliM001.
The third group also combines with the dictionaries, but the weights attributed to the translations are their idf. These runs are prefixed by RaliMidf.
The effectiveness (non-interpolated average precision) of these runs are summarised in the following Globally, we see that the combination with a dictionary (in one of the two ways) leads to some improvement. In particular, for fr-en case, the RaliMidf case is 5.3% better than the RaliP01 case. For de-en, the improvement is even greater (20.7%). This is probably because of the relatively poor coverage of German-English corpus (this corpus is the smallest among all we have mined). On the other hand, there is no observable advantage for the it-en case to combine with a dictionary. Our focus in this CLEF was on fr-en case, because for both French and English, we have a good lemmatizer (in fact, we transform each word in its form of citation). For the two other language, a simple stemmer found from the Web ( http://www.muscat.com , however, the site does not seem to contain the stemmers) was used. For German, this is certainly not enough because of agglutination.
The three best submitted runs are compared with the medium effectiveness in the following table:
Table 9. Comparison with medium effectiveness From this first comparison, we see that the RaliMidfF2E run (the run using a translation model mixed with dictionary translations whose weights are determined by idf) is quite good and well above the medium performance. On the other hand, for Italian and German queries, the performances are quite comparable to the medium performances. We think that the main reason is that we do not have a good stemmer for both languages.
The good effectiveness of the fr-en run shows that it is a good idea to merge a translation model with a bilingual dictionary and to weight the dictionary translations by idf. In so doing, the statistical translation model can be completed by the dictionary, and the idf factor assigns the dictionary translations with a reasonable weight.
In this series of tests on bilingual IR, we wanted to test several ideas:
-the use of cut-offs of statistical translation models, -the clean-up of the training corpora, -the use of two-directional query translation, -and the combination of translation models with bilingual dictionaries. Model cut-off was shown to be not only a way to reduce the space required to store the translation models, but also a way to increase the quality of query translation. This is due to the elimination of many unreliable translations during the cut-offs. In addition, we found that it is better to cut off the models according to the translation probabilities than by size of the model.
However, the clean up of initial training corpora did not show any improvement in the CLIR tasks. This seems to be contradictory with our recent experiments on Chinese-English CLIR. We would like to further examine the processes of our clean up and model training in order to find the reasons for this.
The use of two-directional query translation did not improve the CLIR performance either. Several factors in our implementation may have affected this test, in particular, the setting of translation probability threshold of 0.001 that leads to few translation words per query. Additional tests are undertaken in order to find a more satisfactory explanation.
Finally, the combination of translation models with bilingual dictionaries is, in general, beneficial. In particular, for the case of French-English CLIR that we focused on, we observed a certain degree of improvement. In this case, we found it a good idea to assign a weight to the dictionary translations according to their idf values. This would assign a common translation word with a low weight, and a fairly rare word with high weight. This approach seems to be very intuitive.
We are doing additional test. We hope to be able to report a more thorough account of this series of tests at the workshop.