We present in this report two main approaches to cross-language information retrieval based on the exploitation of multilingual corpora to derive cross-lingual term-term correspondences. These two approaches are evaluated in the framework of the multilingual-4 (ML4) task.
Most approaches to Cross-Language Information Retrieval (CLIR) rely on query
translation based on existing machine-readable dictionaries and/or translation systems ([
We want to experiment here with two methods to exploit parallel corpora for CLIR purposes. The first one relies on the inference of a bilingual semantic representation via cross-language canonical correlation analysis, whereas the second one, more traditional, relies on the extraction of bilingual lexicons from parallel corpora.
However, the CLEF-2003 multilingual collection is not parallel, but comparable,
that is to say that rather than being translations of one another, documents cover the
same topics, in the same domains. Nevertheless, up to now, extraction methods
developed on comparable corpora, unlike methods for parallel corpora, have not provided
results good enough to be directly used in CLIR, as is argued in [
collection itself. The implicit goal behind these experiments is to develop state-of-theart query translation modules, fully adapted to the collection to be searched. 2
As a preprocessing step, we tag and lemmatize corpora, queries and bilingual resources. Only lexical words (nouns, verbs, adverbs, adjectives) are indexed and only single word entries in our resources are used. Our (lexicon-based) lemmatizer provides a partial segmentation for the German compounds. Additionally, we segment German words which were not decomposed by the lemmatizer using the following patterns: Pattern Segmentation A( ˆaeuioy )sB A( ˆaeuioy ) B
A-B A B
3
In this work we automatically model a semantic correspondence between terms of
different languages, in the spirit of cross-lingual latent semantic indexing (CL-LSI) [
For us, the multivariate random variables to which CCA is applied correspond to documentvectors (in the bag of words representation) in English and French, and there is a oneto-one relation between them corresponding to documents that are translations of each other. We will now consider sets of words that are correlated between the two languages (sets of words in the two languages that have a correlated pattern of appearance in the corpus). We will assume that such sets approximate the notion of ’concepts’ in each language, and that such concepts are the translation of each other. Rather than considering plain sets, we will consider terms to have a degree of membership to a given set. In other words, the term will be assigned a weight for each concept we consider, and every concept will correspond to a vector in English, and a vector in French. We will use that weight to form linear combinations of terms, so that they can define a direction in the term space.
Suppose as for CL-LSI we are given aligned texts in, for simplicity, two languages, i.e. every text in one language is a translation of text in another language. In practice, each text can correspond to a complete document, a paragraph, or a sentence. The finer the textual units, the more accurate the correlation statistics. Our hypothesis is that having aligned texts and
we can learn (semantic) directions and where we use the notation so that the projections and of input data images from the different languages would be maximally correlated. These new random variables are univariate, and linear combinations of the previous ones. We consider optimizing this quantity with respect to the choice of and . This leads to the following objective functions and optimization problems:
corr This optimization problem can be transformed into a generalized eigenvalue problem as follows. One is looking for the maximum correlation directions: maximize subject to where we are using the covariance matrix: The solutions of this problem can be obtained by solving a related generalized eigenproblem (1) and the solution directly provides the directions and of maximum correlation:
3.2
The kernel CCA procedure identifies a set of projections from both languages into a common semantic space. This provides a natural framework for performing crosslanguage information retrieval. We first select a number of semantic dimensions, , with largest correlation values . To process an incoming query we expand into the vector representation for its language and project it onto the canonical -correlation components: using the appropriate vector for that language, where is a matrix whose columns are the first solutions of (1) for the given language sorted by eigenvalue in descending order. Notice that in this case we use the standard dot product to perform the projection, but non-linear projections can also be obtained by replacing the dot product with a non-linear kernel ([]). 3.3
The whole training collection consists of 1.3 million pairs of aligned text chunks
(sentences or smaller fragments) from the 36 Canadian Parliament proceedings. We used
only first 1000 documents. The raw text was split into sentences with Adwait
Ratnaparkhi’s MXTERMINATOR and the sentences were aligned with I. Dan Melamed’s GSA
tool (for details on the collection and also for the source see [
The text was split into ’paragraphs’ based on ’***’ delimiters and these ’paragraphs’ were treated as separate documents. After removing stop-words in both French and English parts and rare words (i.e. appearing less than three times) we obtained term-by-document ’English’ matrix and ’French’ matrix (we also removed a few documents that appeared to be problematic when split into paragraphs). The test corpus and queries were processed by Xerox Research Centre Europe as explained in Section 2. The results were unimpressive due to the fact that we restricted ourselves only to the French part - Le Monde - due to the lack of time. Possibly there were also bugs in software and we are working to reveal them, renewed results may appear in the final version. 4
We want to assess in this section the usefulness of bilingual lexicons extracted from collections. In order to illustrate the potential gain this approach could yield, we conducted the following simple experiment. We first collected all the English terms from the queries associated to the ML4 task, from years 2000 to 2003. We then tried to evaluate whether or not we were able to translate those terms with manually built, existing dictionaries, and whether or not we were able to translate them with bilingual lexicons automatically derived from multilingual collections. To this end, we retained two multilingual dictionaries, the ELRA dictionary2, and the Oxford-Hachette3. For corpora, we retained a part of the Hansard4, the JOC corpus (already mentioned in footnote 1, comprising ca. 3.5 millions English tokens), and the ML4 collection itself. For each term present in the set of English queries, we checked whether it was present in the lexicons associated with the above resources. The percentage of English terms found in the lexicons is summarized in table 1.
As one may have noticed, the figures we obtained are only upper bounds on the actual coverage of each resource, since the presence of a term in a dictionary does not imply that the proposed translation(s) are appropriate for the collection at hand. Furthermore, there is an important qualitative difference between manually built and automatically extracted lexicons, a difference that may well balance the advantage for corpus-based methods displayed in table 1. However, were we able to accurately extract bilingual lexicons from corpora, table 1 shows that we would have an important gain over using existing, general purpose dictionaries. Table 2 supports this latter fact and shows how the average precision evolves, on a sub-part of ML4, according to the lexicon used to translate queries.
The column (JOC+ML4) combines the lexicons extracted from the JOC and ML4 corpora, as detailed in section 4.3. The bilingual runs correspond to English queries 2 Multilingual dictionary, available from ELRA, www.elra.info, comprising ca. 45000 English entries 3 Bilingual English-French dictionary, comprising ca. 45000 English entries. 4 In fact a sub-part of it, comprising ca. 20 millions English tokens translated in the corresponding target language (all these experiments, as well as the following ones, are based on the vector-space model). As one can note, the use of automatically derived bilingual lexicons significantly outperforms the use of existing dictionaries on this collection.
We are now going to review the methods we used for extracting bilingual lexicons from parallel and comparable corpora. 4.1
Recent research has demonstrated that statistical alignment models can be highly
successful at extracting word correspondences from parallel corpora ([
We first represent co-occurrences between words across translations by a matrix, the rows of which represent the source language words, the columns the target language words, and the elements of the matrix the expected alignment frequencies (EAFs) for the words appearing in the corresponding row and column. Empty words are added in both languages in order to deal with words with no equivalent in the other language.
The estimation of the expected alignment frequency is based on the Iterative
Proportional Fitting Procedure (IPFP) presented in [
where and are the current estimates of the row and column marginals, is a pair of aligned sentences containing words and , and and are the observed frequencies of words and in . The initial estimates are the observed frequencies of co-occurrences, obtained by considering each pair of aligned sentences and by incrementing the alignment frequencies accordingly. The sequence of updates will eventually converge and the EAFs are then normalized (by dividing each element by the row marginal ), so as to yield probabilistic translation lexicons, in which each source word is associated with a target word through a score. In the remainder of the paper, we will use to denote the probability of selecting target word as translation for source word , as given by this method. 4.2
Bilingual lexicon extraction from non-parallel but comparable corpora has been
studied by a number of researchers, [
Our implementation of this strategy relies on the following steps: 1. For each word , build a context vector by considering all the words occurring in a window centered on , run through the corpus. Each word in the context vector of is then weighted with a measure of its association with . However, in order to ensure we make adequate use of the prior knowledge provided by the general dictionary, we include in its context vector. Lastly, we have used here a window of 5 words before and after , and retained the mutual information as the measure of association. 2. The context vectors of the target words are then translated with our general bilingual dictionary, leaving the weights unchanged (when several translations are proposed by the dictionary, we consider all of them with the same weight) 3. The similarity of each source word , for each target word , is computed on the basis of the cosine measure 4. The similarities are then normalized to yield a probabilistic translation lexicon, . 4.3
Because they contain different information, the comparable and parallel corpora yield different translations that need be combined in order to obtain a complete translated query. Such a combination should account for the fact that for some source words the information provided by the comparable corpus is more reliable than the one provided by the parallel one (as is the case when the source word is not present in the parallel corpus), whereas for some other source words the situation is reversed. However, because of time constraints, we were not able to adopt this strategy, and had to resort to a simpler linear combination, in which the final vector representing the query in target language is given by: (2) is a scalar representing the weight associated with the translation provided by the parallel corpus. We optimized the value of on the queries corresponding to years 2000 to 2002. 4.4
Our strategy to merge results from different languages relies on the fact that if we use “similar” translation matrices, and if the scoring method is identical for each language, then one can directly merge the results from different languages. Using similar translation matrices means that the length (as measured by the norm) of target queries should be identical (since they all issue from the same English query). In order to ensure this, we normalise each target query by its length ( ). Furthermore, to get an equivalent, on the English collection, of the translation step used in the other languages, we consider the English sub-collection to constitue a comparable corpus on its own, from which we build a term-term co-occurrence matrix in exactly the same way as we built a translation matrix in section 4.2 (the source and target languages being identical here). This matrix is then used to expand English queries with most similar terms. 4.5
The results display in table 3 are obtained by translating queries with the
combination of the lexicons derived from JOC and ML4 as explained above. Despite the
important difference the two weighting schemes have on the monolingual collections (cf.
e.g. [
We have tested two main approaches to cross-language information retrieval based on the exploitation of multilingual corpora to derive cross-lingual term-term correspondences. The first approach makes use of parallel corpora to derive an interlingual semantic representation of documents, using canonical correlation analysis. The second approach aims at directly extracting bilingual lexicons, both from parallel and comparable corpora, to be used for query translation. Our experiments show that the second approach outperforms a standard approach using existing bilingual dictionaries for query translation. We plan in the future to pursue the promising road of bilingual lexicon extraction from comparable corpora.
This research was partially supported by the European Commission under the KerMIT Project No. IST-2001-25431.