This paper reports experimental results of cross-language information retrieval (CLIR) from German to French. The authors are concerned with CLIR in cases where available language resources are very limited. Thus transitive translation of queries using English as a pivot language was used to search French document collections for German queries without any direct bilingual dictionary or MT system of these two languages. The two-stage refinement of query translations that we proposed at the previous CLEF 2003 campaign is again used for enhancing performance of pivot language approach. In particular, disambiguation of English terms in the middle stage of transitive translation was attempted as a new trial. Our experiment results show that the two-stage refinement method is able to significantly improve search performance of bilingual IR using a pivot language, but unfortunately, the English disambiguation has almost no effect.
A purpose of the “two-stage refinement technique” is to modify a result of query translation for improving CLIR
performance. The modification consists of two steps: (
We define mathematical notations such that: s j : term in the source query ( j = 1,2,..., m ), T j′ : a set of translations in the target language for term s j , and T = T1′ ∪ T2′ ∪ ... ∪ T ′ .
m
First, the target document collection is searched for the set of terms T . Second, the most frequently appearing term in the top-ranked documents is selected from each set of Tj′ ( j = 1,2,..., m ) respectively. That is, we choose a term ~tj for each Tj′ such as ~tj = arg max rt ( t ∈ T j′ ), ~ ~ ~ ~
T = {t1, t2 ,..., tm}. where rt is the number of top-ranked documents including the term t . Finally, a set of m translations through the disambiguation process is obtained, i.e.,
The disambiguation technique is clearly based on PRF, in which some top-ranked documents are assumed to be relevant. The most frequently appearing term in the relevant document set is considered as a correct translation in the context of a given query.
In the next stage, according to Ballestellos and Croft[2], a standard post-translation query expansion by PRF
technique is executed using T~ in (
(N − nt + 0.5)(R − rt + 0.5)
where N is the total number of documents, R is the number of relevant documents, nt is the number of
documents including term t , and rt is defined as the same as before (see Equation (
The pivot language approach is adopted in this paper, i.e., a search term in the source language is translated into the set of English terms, and each English term is transitively translated into terms in the target language. As many researchers pointed out, if the set of English terms includes erroneous translations, they would yield much more irrelevant terms in the target language.
A solution is to apply any disambiguation technique to the set of English translations (see Fig.1). If an English
document collection is available, we can use easily our disambiguation method described in the previous section.
(
Both German and French texts (in documents and queries) were basically processed by the following steps: (
We downloaded free dictionaries (German to English and English to French) from the Internet1. Also, stemmers and stopword lists for German and French were available through the Snowball project2. Stemming for English was conducted by the original Porter’s algorithm [3].
G to E dictionary
E to F dictionary
two-stage refinement disambiguation expansion original query (German)
translations
document set final French search terms target document set (French)
Before executing transitive translation by two bilingual dictionaries, all terms included in the dictionaries were normalized through stemming and lemmatization processes with the same procedure applied to texts of documents and queries. The actual translation process is a simple replacement, i.e., each normalized German term (to which decomposition process was applied) in a query was replaced with a set of corresponding normalized English words, and similarly, each English word was replaced with the corresponding French words. As a result, for each query, a set of normalized French words was obtained. If no corresponding headword was included in the dictionaries (German-English or English-French), the unknown word was sent directly to the next step without any change.
Next, refinement of the translations by our two-stage technique described in the previous section was
executed. The number of top-ranked documents was set to 100 in both stages, and in the query expansion stage,
the top 30 terms were selected from the ranked list in decreasing order of term weights (Equation (
Let yt be the frequency of a given term in the query. If the top-ranked term was already included in the set of search terms, the term frequency in the query was changed into 1.5 × yt . If not, the term frequency was set to 0.5 (i.e., yt = 0.5 ).
As for dictionary-based transitive query translation via a pivot language, we executed three types of run as follows: - (a) Two-stage refinement of translation with English disambiguation - (b) Two-stage refinement of translation without English disambiguation (same in CLEF 2003) - (c) No refinement
In order to comparatively evaluate performance of our two-stage refinement method, we decided to use commercial MT software produced by a Japanese company3. In this case, first of all, the original German query was entered into the software. The software we used executes automatically German to English translation and then English to French translation (i.e., a kind of transitive translation). The resulting French text from the software was processed according to the procedure described in section 3.1, and finally, a set of normalized French words was obtained for each query. In the case of MT translation, only post-translation query expansion was executed with the same procedure and parameters as the case of dictionary-based translation.
Similarly, for comparison, we tried to execute French monolingual runs with post-translation query expansion.
The well-known the BM25 of Okapi formula [4] was employed for computing each document score in all searches of this study. We executed five runs in which <TITLE> and <DESCRIPTION> fields in each query were used, and submitted the results to the organizers of CLEF 2004. All runs were executed on the information retrieval system, ADOMAS (Advanced Document Management System) developed at Surugadai University in Japan.
The target French collections include 90,261 documents in total. The average document length is 227.14 words. Also, we use the Glasgow Herald 1995 as a document set for English disambiguation. The English collection includes 56,742 documents and the average document length is 231.56. 3 http://www.crosslanguage.co.jp/english/ 0.7 0.6 0.5 0.2 0.1 0
NiiFF01 NiiMt02 NiiDic03 NiiDic04 NiiDic05 is .1015, and NiiDic03 (with English disambiguation) and NiiDic04 (with no English disambiguation) outperform significantly NiiDic05.
However, it looks that the English disambiguation has almost no effect. The MAP score of NiiDic03 is .2690, which is slightly inferior to that of NiiDic04 (.2740), and clearly there is no statistically significant difference between them.
0.0 Recall This paper reported results of our experiment on CLIR from German to French, in which English was used as a pivot language. Two-stage refinement of query translation was employed for removing irrelevant terms in the target language produced by transitive translation using two bilingual dictionaries successively and for expanding the set of translations. Particularly, in CLEF 2004, disambiguation of English terms in the middle process of transitive translation was tried.
As a result, it turned out that − our two-stage refinement method significantly improves retrieval performance of bilingual IR using a pivot language, and − English disambiguation has almost no effect.
Intuitively, the English disambiguation is promising because removing erroneous English term is theoretically effective for preventing irrelevant terms from spreading in the final set of search terms in the target language. Further research is needed.