In this report, we describe the approach we used in CLEF Cross-Language IR (CLIR) tasks. In our experiments, we used statistical models estimated from parallel texts automatically mined from the Web. In our previous experiments, we tested CLIR for English-French and English-Chinese. Our goal of this series of experiments is to see if the approach may be extended to multi-lingual IR (with other languages). In particular, we compare models trained from the Web documents with models that also combine other resources such as dictionaries.
selection of candidate Web sites finding all the documents from the candidate sites paring the texts using simple heuristic criteria
The first step aims to determine the possible web sites where there may be parallel texts for the given language pair. The way we did this is to send requests to some search engines, asking for French documents containing an anchor text such as "English version", "english", and so on; and similarly for English documents. The idea is, if a French document contains such an anchor text, the link to which the anchor is associated usually points to the parallel text in English.
From the set of documents returned by the search engines, we extract the addresses of web sites, which are considered as candidate sites.
The second step also uses the search engines. In this step, a series of requests are sent to the search engines to obtain the URLs of all the documents in each site.
The last step consists of paring up the URLs. We used some heuristic rules to determine quickly is an URL may be parallel to another:
First, parallel texts usually have similar URLs. The only difference between them is often a segment denoting the language of the document. For example, "-en", "-e", and so on for English documents. Their corresponding segments for French are "-fr", "-f", and so on. Therefore, by examining the URLs of the documents, we can quickly determine which files may be a pair.
We then use other criteria such as the length of the file to further confirm or reject a pair. The above criteria do not require to downloading the files actually. Once a set of possible pairs is determined, the paired files are downloaded. Then we can perform some checking of the document contents. For example, are their HTML structures similar? Do they contain enough text? Can we align them into parallel sentences?
The French-English parallel corpus has been constructed last year at RALI laboratoiry. This year, we cooperated with Twenty-One (W. Kraiij) to construct English-Italian and English-German parallel corpora, using the same mining system - PTMiner. The following table shows the number of text pairs as well as volume of the corpora for different language pairs.
Given a set of parallel texts in two languages, it is first aligned into parallel sentences. The criteria used in sentence alignment are the position of the sentence in the text (parallel sentences have similar positions in two parallel texts), the length of the sentence (they are also similar in length), and so on [Gale93]. In [Simard92], it is proposed that cognates may be used as an additional criterion. Cognates refers to the words (e.g. proper names) or symbols (e.g. numbers) that are identical (or very similar in form) in two languages. If two sentences contain such cognates, it provides additional evidence that they are parallel. It has been shown that the approach using cognates performs better than the one without cognates. Before the training of models, each corpus is aligned into parallel sentences using cognate-based alignment algorithm.
Once a set of parallel sentences is obtained, word translation relations are estimated. First, it is assumed that every word in a sentence may be the translation of every word in its parallel sentence. Therefore, the more two words appear often in parallel sentences, the more they are thought of to be translation of one another. In this way, we obtain the initial probabilities of word translation.
At the second step, the probabilities are submitted to a process of Expectation Maximization (EM) in order to maximize the probabilities with respect to the given parallel sentences. The algorithm of EM is described in [Brown93]. The final result is a probability function P(f|e) which gives the probability that f is the translation of e. Using this function, we can determine a set of probable word translations in the target language for each source word, or for a complete query in the source language.
For English and French, we also have other resources: the Hansard corpus (a set of parallel French and English texts from the Canadian parliament debates), a big terminology database (Termuim) and a small bilingual dictionary (Ergane). A translation model is trained from the Hansard data, in the same way as for the Web document (WAC).
In both the terminology database and the bilingual dictionary, we have English words/terms, and their French translations (words/terms). In some way, we can also think of these two resources as two sets of special parallel "sentences". Therefore, the translation probability between words can also be estimated with the same statistical training process. Therefore, two additional translation models are estimated from them. In total, we obtain 4 different translation models between English and French from four different resources (in each direction). The question now is how we can combine them in a reasonable way.
We choose a linear combination of the models. Each model is assigned a coefficient denoting our confidence on it. The coefficient is tuned according to a set of "held-out" data - a set of parallel sentences (about 100K words). This set is selected from different resources (however different from the resources we are using for model training) so that it gives a good balance of different kinds of texts. Finally, the following coefficients are assigned to each model:
0.0413916 0.300517 0.413493 0.244598
As we can see, the combination seems to favor the model which contains more vocabulary. Termium is attributed the highest coefficient because it contains about 1 million words/terms in each language. The Hansard corpus and the Wac corpus contain about the same volume of texts. So their coefficients are comparable. The Ergane dictionary is a small dictionary which only contains 9000 words in each language, its coefficient is very low. Although these coefficients are the best for the held-out data, they may not be suitable to our data in CLEF.
We used a modified version of SMART system [Buckley85] for monolingual document indexing and retrieval. The ltn weighting scheme is used for documents. For queries, we used the probabilities provided by the probabilistic model, multiplied by the idf factor. From the translation words obtained, we retained the top 50 words for each query. The alue of 50 seemed to be a reasonable number on TREC6 and TREC7 data.
Monolingual IR results have been submitted for the following languages: French, Italian and German. This series of experiments uses the SMART ltn weighting scheme for queries as well. In addition, a pseudo-relevance feedback is applied, which uses the 100 most important terms among the top 30 documents retrieved to revise the original queries. The parameters used for this process is: α = 0.75, and β = 0.25. The results obtained are shown below:
For French, we used a lemmatizer developed in the RALI laboratory that first uses a statistical tagger, then transforms a word to its citation form according to its part-of-speech category. For Italian and German, two simple stemmers obtained from the Web [Stemmers] are used. There is no particular processing for compound words in German. This may be an important factor that affected the effectiveness of German IR.
Overall, the French and Italian monolingual runs seem to be comparable to the medium performance of the participants; but the German run is well below the medium performance. We think the main reason is due to the lack of special processing on German (e.g. compound words).
The bilingual task consists of finding documents in a language different from that of the queries. We tested the following bilingual IR: E-F (i.e. English queries for French documents), E-I and E-G. For this series of test, we first used the translation models to obtain a set of 50 weighted translation words for each query. Unknown words are not translated, and they are added into the translation words with the default probability of 0.05. The same pseudo-relevance feedback process is used.
Between English and Italian, English and German, we only have the Web parallel documents to train our translation models. For French and English, we have multiple translation resources: the Web documents, the Hansard corpus, and two bilingual dictionaries. So we also compare the model with only the Web documents (the WAC model) and the model with all the resources combined (the Mixed model). The following table summarizes the results we obtained for the official submissions in bilingual IR. WAC
Mixed 20 13 16 17 21 13
For F-E and I-E cases, the WAC models lead to an effectiveness that is better than the medium. The Mixed model of F-E gives a medium performance. The comparison between the two translation models for French to English is particularly interesting. We expected that the Mixed model could perform better because it is trained with more data from difference sources. Surprisingly, its effectiveness is worse than the Wac model. There may be several reasons to this:
The combination of different resources is tailored for a set of held-out data that does not come from the CLEF document set. So there may be a bias in the combination.
During the combination, we observed that the combination results tend to favor dictionary translations. A high priority is attributed to dictionary translations. This may also be attributed to the biased tuning of combination.
The unreasonable combination may be further illustrated by the following examination of the bilingual IR effectiveness with each individual translation model. The following table shows the effectiveness over all the 40 queries for bilingual IR between French and English:
In our case, the multilingual runs are only possible from English to all the languages (English, French, Italian and German). In these experiments, the following three steps have been done: 1. Translate English queries to French, Italian and German, respectively; 2. Retrieve document from different document sets;
Merge the retrieval results.
The translation of English queries to German and Italian was done by the WAC translation model (trained from the Web documents). For English to French, we also have the alternative of using the Mixed model. The translation words are submitted to the mtc transformation of SMART. This scheme is chosen because it leads to comparable similarities between different data sets, therefore, makes the result merging easier. The merging is done according to the similarity scores. The top 1000 retrieved are selected as the final results and submitted for evaluation.
The following table describes the results of different runs. In the Wac column, all the models used to translate English queries are WAC models. In the Mixed case, only the English to French translation uses the Mixed model, whereas the other translations still use the Wac models. ≥ medium 14 26 12 28
As we can see, these performances are all below the medium performance. One of the main reasons may be that the German monolingual retrieval does not use any linguistic preprocessing, and has a very poor effectiveness. This may greatly affected the multilingual runs. Another possible reason may be the over-simplified merging method we used. In fact, in order to render the English monolingual runs compatible (in terms of similarity values) with other bilingual runs, we had to choose the mtc weighting scheme as for the other cases. In our tests, we observe that this weighting scheme is not as good as ltc. Therefore, the ease of result merge has been obtained to the detriment of English effectiveness.
We observe again the negative impact of the Mixed model in this task. When the Wac model for English-French is replaced by the Mixed model, the effectiveness decreases. This shows once again that the coefficients we set for different models are not suitable for the CLEF data.
In this CLEF, we successfully used parallel Web pages to train translation models for language pairs other than English and French.
For monolingual IR, the effectiveness for French and Italian are similar to the medium performance. The German monolingual run is well below the medium. We think the main reason is that we did not carry out any particular processing on German morphology, which is an important problem for German IR.
For bilingual IR between English and French, and between English and Italian, the effectiveness seems to be reasonable. It is better than the medium effectiveness. Between English and German, however, the effectiveness is well below the medium effectiveness. The reason may be the same as for the German monolingual run.
For multilingual runs, the performance is below the medium. We believe the reason is once again the low effectiveness for German. In addition, result merging may also have affected the global effectiveness.
Between English and French, we also tried to combine different resources in our translation models. We used a linear combination of the models trained with different data, and the coefficients are determined by using a small set of held-out data. However, to our surprise, the mixed model performed worse than the model trained with the Web documents only. This clearly indicates that the combination is not reasonable to the CLEF data.
In our future work, we will try to determine a better way to combine different translation models between English and French. For German, we will use more linguistic processing, in particular, a more sophisticated stemmer.
Overall, we are still encouraged by this CLEF because we showed for the fist time that the Web parallel documents could be used for multilingual IR. [Stemmers] http://www.muscat.com