This report describes our test on using statistical translation models for bilingual IR tasks in CLEF-2001. These translation models have been trained on a set of parallel web pages automatically mined from the Web. Our goal is to compare the following approaches: using the original parallel corpora or a cleaned corpora to train translation models; using the raw translation probabilities to weigh query words or combine the probabilities with IDF; using different cut-off probability values in the translation models (i.e. delete the translations lower than a threshold).
the clean up of the parallel corpora in order to create neater corpora; the cut-off of translation models in order to eliminate unreliable translations; the use of two-directional query translation; and the combination of translation models with bilingual dictionaries to increase the coverage.
Given a set of parallel texts in two languages, it is first aligned into parallel sentences. The criteria used in sentence alignment are the position of the sentence in the text (parallel sentences have similar positions in two parallel texts), the length of the sentence (they are also similar in length), and so on [Gale93]. In [Simard92], it is proposed that cognates may be used as an additional criterion. Cognates refers to the words (e.g. proper names) or symbols (e.g. numbers) that are identical (or very similar in form) in two languages. If two sentences contain such cognates, it provides additional evidence that they are parallel. It has been shown that the approach using cognates performs better than the one without cognates. Before the training of models, each corpus is aligned into parallel sentences using cognate-based alignment algorithm.
Once a set of parallel sentences is obtained, word translation relations are estimated. First, it is assumed that every word in a sentence may be the translation of every word in its parallel sentence. Therefore, the more two words appear often in parallel sentences, the more they are thought of to be translation of one another. In this way, we obtain the initial probabilities of word translation.
At the second step, the probabilities are submitted to a process of Expectation Maximization (EM) in order to maximize the probabilities with respect to the given parallel sentences. The algorithm of EM is described in [Brown93]. The final result is a probability function P(f|e) which gives the probability that f is the translation of e. Using this function, we can determine a set of probable word translations in the target language for each source word, or for a complete query in the source language.
A translation model contains translations for every lexical item encountered in a training corpus, even if the item appeared only once. In the case where an item does not appear often, its translation is often sparse, i.e. it is translated by many different words with low probabilities. In fact, if a source word occurs only once in the training corpus, it may be translated by every word in the target sentence with quite comparable probabilities. Such translations are not reliable. In addition, storing the translations for every word requires a large space. It is not practical to use such a translation model in a real application. For example, the original French-English model contains more than 13 millions entries. Therefore, we exploited the possibility of reducing the translation models by cutting off the translations that we judge unreliable.
Two ways to cut off a model are investigated:
We tested several numbers in the first cut-off method: 1 million (1M), 100 thousands (100K), 10 thousands (10K), 5 thousands (5K) and 1 thousand (1K), and the thresholds tested are 0.05, 0.1 and 0.25. Below is a summary of the results with the CLEF-2000 data1. 1 Note: The experimental results on fr-en are missing. We are redoing the experiments to complete the missing numbers and will report them at the workshop. 0.1693 0.2444 0.1651 0.2524 0.1640 0.2393
The parallel corpora gathered from the Web contain a certain proportion of non-parallel texts. In fact, it is impossible to eliminate all the non-parallel texts in such a corpus automatically. The question we raised is whether it is helpful to use additional criteria to filter the corpora so that certain non-parallel texts can be eliminated. We did this filtering first on the Chinese-English corpus, also gathered with the same mining tool. The filtering criteria are the following [Nie01]: Control on empty alignment: Once an automatic alignment algorithm is applied on a pair of texts, if the proportion of empty alignments (a sentence is aligned with nothing, or n:0 and 0:n alignments) is higher than a threshold, then the pair is eliminated.
Using a bilingual dictionary in alignment: In order to increase the accuracy of sentence alignment, we also examine the proportion of words in the sentence that have known translations (stored in the dictionary) in the corresponding sentence. This proportion is taken into account and used in combination with the length criterion in a length-based alignment algorithm [Gale93].
In our tests on Chinese-English corpus, we found that after the filtering, both the translation accuracy judged with 200 randomly selected words and the effectiveness of CLIR between English and Chinese have been increased significantly. Below are the best improvements we have been able to obtain:
161 (80.50%) 154 (77.00%)
We expected to have similar results with the same filtering on the corpora of European languages. However, our tests showed a decrease in performances. That we can see by comparing the following table with Tables 2 and 3. There may be a flaw in the training of the translation models, as this was done in a hurry in the last minutes.
We are now investigating both factors, and hope to find an explanation soon.
It is observed that certain common English words often appear as one of the top-raked translations of queries (e.g. make, provide, due, etc.). This is because these words frequently appear in the training corpora, and they are not considered as stopwords. So they are considered to be highly probable translations for many words in French, German and Italian. However, if we also consider translations of these English words back to the source languages, it would likely be translated to many different words, i.e. their translations would not be concentrated on the same source words for which they have been suggested as translations. Therefore, a combination of both directions in translation could eliminate such common words in query translations. This is the intuition of using the two-direction translation of queries.
In our implementation, the translation probabilities of both directions are multiplied, i.e. for a pair (e, f) of English and French, the probability of P(e|f) is multiplied by the probabilities of Σf' P(f'|e) where f' is all the words in the original French query.
However, our test did not show that this idea or this implementation works well. The following table summarises the results: de-en it-en fr-en 1M 0.1026 0.2116 100K 0.1337 0.2149 10K 0.1339
We observe a large decrease in performance compared with Tables 2 and 3. A possible reason is the filtering of the translations: after the translation of each query, we only keep the translations whose probability is higher than a certain threshold (0.001). So in many cases, there are much less than 50 translation words (as in the case of one-directional translations). As a matter of fact, the average numbers of English translation words per query are about 14 for French queries, less than 17 for German queries and less than 16 for Italian queries. This shows that the way we used to keep translation words is too selective. As an example, the first Italian query (on "reconstruction of Berlin") is translated by the following words: berlin cad part rebuild city wall architecture translation. We will experiments some variances of this method in order to know if it is the idea or our particular implementation method that did not work well.
In the past, we have found that we can greatly improve the CLIR effectiveness by combining a translation model with a bilingual dictionary. The method we tested is to assign a fixed value to those translations that are found in the dictionary. In other words, if a translation is stored in the dictionary, then the weight of this translation will be incremented by a fixed value.
We observed that common words often have more translations than specialised words. As a result, common words will be represented by more translations, and indirectly, their importance in the query is increased unduly. In order to take into account the commonness of a translation word, we combine the translation probabilities with the idf value of the translation word. The intuition is to increase the probabilities of the uncommon translation words and lower the probabilities of common translation words.
We submitted 9 bilingual runs, three for each of the language pairs: fr-en, de-en and it-en.
The first group of runs uses only the 50 first translation words and the translation probabilities. The translation models used are cut-off by a threshold of 0.1 (P≥0.1). These runs are prefixed by RaliP01.
The second group combines the models used in the previous group with dictionaries - FreeDict that we found on the Web (http://www.freedict.com/dictionary/index.html). In particular, we used the dictionaries of French to English, German to English and Italian to English. All the translations in the dictionaries are assigned a fixed value (0.001). These runs are prefixed by RaliM001.
The third group also combines with the dictionaries, but the weights attributed to the translations are their idf. These runs are prefixed by RaliMidf.
The effectiveness (non-interpolated average precision) of these runs are summarised in the following table: fr-en de-en it-en
Globally, we see that the combination with a dictionary (in one of the two ways) leads to some improvement. In particular, for fr-en case, the RaliMidf case is 5.3% better than the RaliP01 case. For de-en, the improvement is even greater (20.7%). This is probably because of the relatively poor coverage of GermanEnglish corpus (this corpus is the smallest among all we have mined). On the other hand, there is no observable advantage for the it-en case to combine with a dictionary.
Our focus in this CLEF was on fr-en case, because for both French and English, we have a good lemmatizer (in fact, we transform each word in its form of citation). For the two other language, a simple stemmer found from the Web (http://www.muscat.com, however, the site does not seem to contain the stemmers) was used. For German, this is certainly not enough because of agglutination.
The three best submitted runs are compared with the medium effectiveness in the following table:
RaliM001D2E ≥ medium < medium 41 6 27 20 27 20
From this first comparison, we see that the RaliMidfF2E run (the run using a translation model mixed with dictionary translations whose weights are determined by idf) is quite good and well above the medium performance. On the other hand, for Italian and German queries, the performances are quite comparable to the medium performances. We think that the main reason is that we do not have a good stemmer for both languages.
The good effectiveness of the fr-en run shows that it is a good idea to merge a translation model with a bilingual dictionary and to weight the dictionary translations by idf. In so doing, the statistical translation model can be completed by the dictionary, and the idf factor assigns the dictionary translations with a reasonable weight. In this series of tests on bilingual IR, we wanted to test several ideas: the use of cut-offs of statistical translation models, the clean-up of the training corpora, the use of two-directional query translation, and the combination of translation models with bilingual dictionaries.
Model cut-off was shown to be not only a way to reduce the space required to store the translation models, but also a way to increase the quality of query translation. This is due to the elimination of many unreliable translations during the cut-offs. In addition, we found that it is better to cut off the models according to the translation probabilities than by size of the model.
However, the clean up of initial training corpora did not show any improvement in the CLIR tasks. This seems to be contradictory with our recent experiments on Chinese-English CLIR. We would like to further examine the processes of our clean up and model training in order to find the reasons for this.
The use of two-directional query translation did not improve the CLIR performance either. Several factors in our implementation may have affected this test, in particular, the setting of translation probability threshold of 0.001 that leads to few translation words per query. Additional tests are undertaken in order to find a more satisfactory explanation.
Finally, the combination of translation models with bilingual dictionaries is, in general, beneficial. In particular, for the case of French-English CLIR that we focused on, we observed a certain degree of improvement. In this case, we found it a good idea to assign a weight to the dictionary translations according to their idf values. This would assign a common translation word with a low weight, and a fairly rare word with high weight. This approach seems to be very intuitive.
We are doing additional test. We hope to be able to report a more thorough account of this series of tests at the workshop.