This paper reports on the participation of ITC-irst in the Cross Language Evaluation Forum 2003; in particular, in the monolingual, bilingual, small multilingual, and spoken document retrieval tracks. Considered languages were English, French, German, Italian, and Spanish. With respect to our CLEF 2002 system, the statistical models for bilingual document retrieval have been improved, more languages have been considered, and a novel multilingual information retrieval system has been developed, which combines several bilingual retrieval models into a statistical framework. As in the last CLEF, bilingual models integrate retrieval and translation scores over the set of N-best translations of the source query.
The statistical cross-language information retrieval
(CLIR) model presented in the 2002 CLEF
evaluation
Let a multilingual collection D contain documents in L different languages, where D results from the union of L monolingual sub-collections D1; : : : ; DL. Let f be a query in a given source language, eventually different from any of the L languages. One would like to rank documents d within the multilingual collection D, according to the posterior probability:
Pr(d j f ) / Pr(f ; d) where the right term of formula (1) follows from the constancy of Pr(f ), with respect to the ranking of documents.
A hidden variable l is introduced, which represents the language of either a sub-collection or a document.
Pr(f ; d) ==
X Pr(l; f ; d) l l =
X Pr(l) Pr(f ; d j l) where Pr(l) is an a-priori distribution over languages, which can be estimated from the multilingual collection or taken uniform. Formula (2) shows a weighted mixture of bilingual IR models depending on the subcollection. However, given that we know the language each document is written in, we can assume (1) (2) that the probability Pr(f ; d j l) is larger than zero only if d belongs to the sub-collection Dl.
Next, a hidden variable e is introduced, which represents a (term-by-term) translation of f into one of the L languages. Hence, we derive the following decomposition:
Pr(f ; d j l) =
X Pr(f ; e; d j l)
e ¼
X Pr(f ; e j l) Pr(d j e; l) (3)
e In deriving formula (3), we make the assumption (or approximation) that the probability of document d given query f , translation e and language l, does not depend on f . Formula (3) puts in evidence a language-dependent query-translation model, Pr(f ; e j l), and a collection-dependent query-document model, Pr(d j e; l).
The language-dependent query-translation model is defined as follows: Pr(f ; e j l) = Pr(f j l)Prl(e j f ) 8 Prl(f ; e) >>>> X <
Prl(f ; e0) /
e02Tl(f ) > > > > :0 if e 2 Tl(f ) otherwise where Tl(f ) is the set of all translations of f into language l. For practical reasons, this set is approximated with the set of the N most probable translations computed by the basic query-translation model Prl(f ; e). The term Pr(f j l) can be considered independent from l and hence be discarded. The normalization introduced in formula (4) is needed in order to obtain ranking scores, which are comparable among different languages.
The collection-dependent query-document model is derived from a basic query-document model Prl(d j e) as follows: 8 Prl(d; e) >>>> X < Pr(d j e; l) = d02I(e;l)
Prl(d0; e) > > > > :0 if d 2 I(e; l) otherwise where I(e; l) is the set of documents in Dl containing at least a word of e.
The basic query document and query translation
models are now briefly described; more details can
be found in
Language Model The joint probability can be factored out as follows:
Pr(e; d) = Pr(e j d) Pr(d) where the a-priori probability of d, Pr(d), is assumed to be uniform, and the probability of e given d to be an order-free multinomial (bag-of-word) model: n Pr(e = e1; : : : ; en j d) = Y p(ek j d) k=1 Okapi The joint probability can be obtained through the normalization over queries and documents of a generic scoring function s(e; d): Pr(e; d) = Pe0;d0 s(e0; d0) s(e; d) The denominator is considered only for the sake of normalization, but can be disregarded in the computation of equation (3).
A scoring function derived from the standard Okapi
formula, is used
s(e = e1; : : : ; en; d) =
n
Y idf (ek)Wd(ek)
k=1
(4)
(5)
(6)
(7)
(8)
Combination Previous work
n n
= p(e1) Y p(ek j ek¡1) Y p(fk j ek)
k=2 k=1
The term translation probabilities p(f j e) are
estimated from a bilingual dictionary as follows:
Pr(f j e) = Pf0 ±(f 0; e)
±(f; e)
(9)
where ±(f; e) = 1 if the term e is one of the
translations of term f and ±(f; e) = 0 otherwise. This flat
distribution can be refined through the EM algorithm
The target LM probabilities p(e j e0), are estimated on the target document collection, through an orderfree bigram LM, which tries to compensate for different word positions induced by the source and target languages. Let p(e j e0) = Pe00 p(e00; e0) p(e; e0) (10) where p(e; e0) is the probability of e co-occurring with e0, regardless of the order, within a text window of fixed size. Smoothing of this probability is performed through absolute discounting and interpolation.
3.
As shown in Section 2, the ITC-irst multilingual IR system features several independent bilingual retrieval systems, which return collection-dependent rankings, and a module for merging these results into a global ranking with respect to the whole multilingual collection. Moreover, language-dependent text preprocessing modules have been implemented to process documents and queries. Figure 3. shows the architecture of the system.
Two merging criteria were developed. The first, we call stat method, implements the statistical model introduced in Section 2: for each language, languagedependent relevance scores of documents, computed by the bilingual IR systems are normalized in order to have language independent scores, and, hence, a global ranking is created.
The second criterion, we call rank method, exploits the document rank positions only, i.e. all the collection dependent rank lists are joined and documents are globally sorted according to the inverse of their original rank position.
Monolingual and bilingual versions of the system trivially follows by omitting the query-translation model and by limiting the collection to one language, respectively.
In order to homogenize the preparation of data, and, hence, to reduce workload, a standard procedure was defined. More specifically, the following preprocessing steps were applied both to documents and queries in every language: Documents Bilingual Dictionary
Bilingual IR
Bilingual IR Documents Bilingual
Dictionary Query (source) Preprocessing
Merging
Ranked Documents
² Tokenization was performed to separate words
from punctuation marks, to recognize
abbreviations and acronyms, correct possible word splits
across lines, and discriminate between accents
and quotation marks.
² Stemming was performed by using a
languagedependent Porter-like algorithm
After document ranking, the following Blind
Relevance Feedback (BRF) technique was applied. First,
the documents matching the source query e are
ranked, then the B best ranked documents are taken
and the R most relevant terms in them are added to
the query, and the retrieval phase is repeated. In the
CLIR framework, R terms are added to each single
translation of the N -best list and the retrieval
algorithms is repeated once again. In this work, 15 new
search terms are selected from the top 5 documents
according to the Offer Weight proposed in
4.
ITC-irst submitted 4 monolingual runs in French, German, Italian, and Spanish, 4 Italian-Spanish bilingual runs, 2 German-Italian bilingual runs, and 4 small multilingual runs using queries in English to search documents in English, French, German, and Spanish. Moreover, some unofficial experiments were performed for the sake of comparison. 4.1.
In Table 1, statistics about the target collections for the five considered languages are reported.
English French German Italian Spanish Multi-4
Bilingual dictionaries from English to the other languages were gathered from public available resources. Unfortunately, German-Italian and ItalianSpanish dictionaries were not available. Hence, the missing dictionaries were built from other available dictionaries using English as a pivot language. For example, an Italian-Spanish dictionary was derived by exploiting the Spanish-English and ItalianEnglish dictionaries as follows: the translation alternatives of an Italian term are all Spanish translations of all English translations of that term. Table 2 reports some statistics of the bilingual dictionaries. It is worth noticing that for the generated dictionaries the average number of translation alternatives is about twice larger than that of original dictionaries. This would suggest that they contain two wrong translations per entry, on the average.
Moreover, all term translation probabilities, but the German-Italian ones, were estimated through the EM algorithm by using the corresponding document collections. 4.2.
Monolingual results As shown in Table 4, our monolingual retrieval system achieves good results for all languages. More than 70% of queries have mAvPr greater than or equal to the median values. It is worth noticing that mAvPrs are pretty the same for all languages.
Bilingual results Italian-Spanish results show that the estimation of translation probabilities through the EM algorithm is quite effective, especially in combination with the 10-best translations.
Spanish
Multilingual results As shown in Table 4, about 60% of the queries have mAvPr greater than or equal to the median values. The merging method based on the rank is a little more effective, but differences are very low. Again, the EM estimation of term probabilities slightly improves performance.
The merging criteria were also applied to the monolingual runs, in order to obtain an upper bound for our multilingual retrieval system. The achieved mAvPrs for this virtual experiment were .3754 and .3667 for the “rank” and “stat” criteria, respectively. The relative degradation is very similar to that observer for bilingual experiments.
Cross-Language Spoken Document
Retrieval ITC-irst participated also in the Cross-Language Spoken Document Retrieval (CLSDR) track, which consists in searching for relevant stories within a collection of automatically transcribed English broadcast news. Topics correspond in 50 short queries manually translated from English into French, German, Italian, and Spanish. For the CLSDR track, the bilingual version of the ITC-irst IR system was applied, with little changes in the BRF expansion of queries. Moreover, German text were also processed for splitting compound words, by using a DP based algorithm. 5.1.
Query expansion on parallel corpora As the number of stories in the SDR target collection was quite small, a double query expansion policy was chosen. New terms are added which are extracted not only from the target collection, but also from a large corpus of written texts, consisting of newspapers and news wires.
As a parallel corpus for query expansion, newspaper articles of the North American News Text corpus were used (www.nist.gov/speech/tests/sdr). In particular, 313K documents are extracted from Los Angeles Times, Washington Post, New York Times, and Associated Press Worldstream, issued between September 1997 and April 1998. Unfortunately, the available texts do not entirely cover the test period. The following strategy was chosen: first query expansion was performed on parallel texts, and then on target collection.
Table 6 reports the official submitted runs, and some unofficial runs (in italics), used for comparison.
Official run mono-brf mono-brf-brf fr-en-1bst-brf-bfr fr-en-sys-brf-bfr de-en-dec-1bst-brf-bfr de-en-1bst-brf-bfr de-en-sys-brf-bfr it-en-1bst-brf-bfr it-en-sys-brf-bfr es-en-1bst-brf-bfr es-en-sys-brf-bfr
EN .3944 EN .4244 FR .2281 FR .3064 DE .2676 DE .2523 DE .2880 IT .2347 IT .3218 ES .2746 ES .3555 The official English monolingual run was performed in order to evaluate the quality of the retrieval system. ITC-irst performance is about 10% above the other participants. For this experiment the query expansion on the parallel corpus was not applied. If not so, a relative improvement of 7% is observed. As the double query expansion policy is quite effective, was applied in all the other experiments.
In the bilingual experiments, query were translated either through our 1-best translation approach or by the Babelfish translation service, powered by Systran, which is available on the Internet (world.altavista.com). Run names are indicating with 1bst and sys, respectively. Commercial translations outperforms our approach.
German word decompounding seems to be slightly effective, as shown by comparing the run without decompounding ( de-en-1bst-brf-bfr) and the with (de-en-dec-1bst-brf-bfr).
6.
This paper presented a multilingual IR system developed at ITC-irst. A complete statistical model was defined which combines several bilingual retrieval model. The system was evaluated in the CLEF2003 campaign in the monolingual, bilingual, and multilingual tracks. The basic monolingual IR model resulted very competitive on every languages. The multilingual IR systems also achieves higher performance than the median. Experiments in the Cross-Language Spoken Document Retrieval task, which uses very short queries, showed that significantly better results are still achieved by using translations produced by a commercial system.
7.