Choosing the correct target words is a difficult problem for machine translation. In cross-language information retrieval, this problem of choice is mitigated since more than one translation alternative can be retained in the target query. Between choosing just one word as a translation and keeping all the possible translations for each source word, one can apply a range of filtering techniques for eliminating some words and keeping others. In the bilingual track of CLEF 2002, focusing on word translation ambiguity, we experimented with several techniques for choosing the best target translation for each source query word by using co-occurrence statistics in a reference corpus consisting of documents in the target language. One of two distinct corpora was used, the target-language test corpus or the World Wide Web. Our techniques give one best translation per source query word. We also experimented with combining these word choice results (providing up to three translations for each word) in the final translated query. The source query languages were Spanish and Chinese; the target language documents were in English. We submitted four automatic runs for each language pair. When the methods were combined, mixing results obtained with different reference corpora, the recall and average precision of Spanish-to-English retrieval reached 95% and 97%, respectively, of the recall and average precision of an English monolingual retrieval run. For Chinese-to-English text retrieval, the recall and average precision reached 89% and 60%, respectively, of the English run.
Choosing among lexical or translation variants to find correct target words is a difficult problem for machine
translation. In cross-language information retrieval (CLIR), the problem of choosing a single best translation is
less critical in theory since more than one translation alternative can frequently be retained in the target query
with only minimal harm to retrieval performance. However, there is still a range of possible performance
tradeoffs between choosing just one word as a translation and keeping all the possible translations for each
source word
For cross-language information retrieval, both the documents and the queries need to be represented in the same
language at some po
Spanish queries were processed as follows. The text of the Spanish query was tokenized and morphologically analyzed using a Spanish version of Clarit. Only nouns, verbs, adjectives, and adverbs were retained for further treatment. Some additional words were removed via a stop list containing a total of 400 words. This list includes all prepositions, pronouns, and articles (which were already removed using the morphological analyzer); common stop words such as “es”, “cada”; and query meta-language from previous CLEF queries such as “describir” and “discutir”.
As an example of our processing consider the Spanish query on the Leaning Tower of Pisa (Topic 136): Torre
inclinada de Pisa. ¿En qué estado se encuentra la torre inclinada de Pisa? After morphological analysis and
stop word removal, this query becomes torre, inclinar, pisa, estado, torre, inclinar, pisa. Each of these words is
then looked up in a Spanish–English word-to-word dictionary, which contains the following translations:
inclinar
apt
bow
drooping
incline
inclined
inclining
sloping
stooping
titling
verging
estado
state
states
statis
pisa
pisa
torre
high
tower
towers
We created our Spanish-English gloss lexicon by combining various lexicons available on the Web. The final
collation was not manually edited; stop words were automatically removed from the English translations (unless
the only translation was a stop word); and the Spanish side of the dictionary was lemmatized (e.g., an original
gloss such as “inclinado—apt to” was reduced to “inclinar—apt” in our experimental version). If a source word
was not found in the dictionary then the original source word was retained in lieu of a translation. The resulting
translations formed the basis of the English queries that were generated by the methods described below.
We can note here two limitations of our technique: (1) The dictionaries are neither clean nor complete. Notice
that “leaning” is missing from the above translations of “inclinar”. (2) We have restricted ourselves to
word-toword translations for engineering reasons, even though we know that phrasal translation results in superior
performance in CLIR
Since spaces are not used in Chinese text for word segmentation, we first broke Chinese text into individual words. We used the longest-match method, which greedily recognizes an initial string of characters as a word if the string matches a word in the segmentation dictionary. We obtained a Chinese-to-English wordlist from the Linguistics Data Consortium1 (LDC). This bilingual wordlist contains a list of Chinese words together with their possible English translations—a total of 188,474 entries. We did not edit or further “clean” the lexicon. Our run-time segmentation dictionary consisted of the Chinese words from this wordlist augmented with all possible single Chinese characters and symbols. By using the words from the bilingual wordlist, we ensured that the words identified during segmentation would have glosses during translation.
Once we obtained the segmented Chinese words, we first removed stop words automatically via a stop word list. The list contains a total of 3,894 entries, including closed-class words (e.g., symbols, prepositions, pronouns, particles), numerals, and query specific terms such as and , collected from CLEF 2001 topics. Then we translated the remaining words into English using the LDC bilingual wordlist. Lastly, we used the translation disambiguation methods (described below) to select the best translation for a query word. 1 http://www.ldc.upenn.edu/Projects/Chinese/LDC_ch.htm#e2cdict As an example of our processing, consider the Chinese version of the Leaning Tower of Pisa query (Topic 136): ; First the query is segmented as: ; ; ; ; ; ; ; ; ; ; ; After the stop-word removal, the query contains unique words: ; ; ; ; . Each of these words is then looked up in the Chinese–English bilingual wordlist, which yields the following translations: pisa askant slanting ter pagoda tower hygeia exuberance health healthiness circumstance circumstantiality situation state affairs We should note that there are several problems with the longest-match method that can cause segmentation errors. First, the greedy algorithm may break words in the wrong places when word boundaries are ambiguous. Generally, a wrong segmentation will result in more subsequent segmentation errors. Second, the coverage of the dictionary will affect segmentation quality. Missing dictionary words will result in single characters being generated during segmentation. For bilingual retrieval, this not only reduces term quality, but also increases ambiguity, since single characters are generally more ambiguous than multiple character words. This is especially a problem with proper names, where the meanings of the single characters in the names bear no relation to the meaning of the name. In the pre-processing of topics, we eliminated any sequence of more than three consecutive single characters. This helped reduce translation noise, but without proper handling of the names, we still lost the specific information associated with deleted characters, which tended to render the topics too general. 3
Given that a source language term can give more than one possible target translation, we want to find the best or the best few translations for each non-stop word in the source topic. Our methods for disambiguating alternative possible translations are based on two observations:
The correct translation for the query term, given its potential translations in a target language, is generally not ambiguous when context (i.e., other terms in the query) is considered.
The Web and reference corpora can be used as practical resources for estimating the coherence of the translated terms. Each provides a language model of how words co-occur. In particular, we expect that words that are found to co-occur are lexically-semantically cohesive.
For example, suppose the query terms s1,…,s5 in the source language have the translations in the target language as follows: • • s1 t11 t12 t13 s2 t21 t22 s3 t31 t32 t33 t34 s4 t41 <t13, t22, t34, t41, t52> Term s1 has three possible translations: t11, t12, and t13. A context for t11 can be constructed as one of the possible sequences including the other translations in the target language, such as, In this example, there are a total of 3 × 2 × 4 × 1 × 2 (i.e., 48) possible sequences or paths through the translation space. Each path establishes a context for the translated terms with respect to their neighbors. We assume that the best path of all combinations will demonstrate the best coherence among the translated terms. We have developed several practical methods to measure the quality (or coherence) of translation paths based on evidence of actual word co-occurrence in reference corpora. One of these methods takes advantage of the World Wide Web (WWW) and two use the actual target test collection, as described in the following sections.
The Web method is an elaboration of the ideas explored by Grefenstette (1999), namely, using the WWW as the
language model for choosing translations. The idea of using the Web to acquire general language models is
becoming more popular.
4 Collate the selected translations into a new target language query.
Since word order is not preserved from one language to another, the query sent to the Web uses an operator that enforces presence of the translated sequence but not sequence order. AltaVista’s advanced search supports the operator NEAR, which ensures that the words so linked appear within ten words of each other, in any order. We use this operator to calculate the score of the sequence.
Since the Web searches do not stem search terms, we expanded each translation by linking all surface forms of the search term by OR. For example, if the translated sequence being scored was “big black dogs” then the following, advanced AltaVista query was generated: (big OR bigger OR biggest) NEAR (black OR blacks OR blacked OR blacking OR blackest OR blacking) NEAR (dog OR dogs) The Web method has the advantage of a massive reference corpus: most candidate translations (paths) will result in some “hits” and the number of hits for meaningful combinations of words will typically be much greater than for meaningless ones. It has the potential disadvantage that the texts on the Web may bear little or no direct relation to the texts (or domain) of the target search. In theory, a narrowly focused target search (e.g., in a technical domain reflected by many documents concentrated in the database to be searched) might be underrepresented in the Web corpus compared to alternative, more common documents. To mitigate this possibility, we also explored two methods that use only the target texts for co-occurrence evidence.
An alternative reference corpus for language modeling, particularly modeling the coherence of combinations of translation alternatives, is the target corpus itself. We use the target corpus as the basis for choosing a “best” translation of a query, exploiting approaches developed by Evans (2000; 2001). We implemented two targetcorpus methods (“Corpus1” and “Corpus2”). 3.2.1 Corpus1 The Corpus1 method has the following steps: 1 Get translations for each term in the query. 2 Construct a hypothesis space of translated queries by obtaining all possible combinations of the translations. 3 For each translated sequence in the hypothesis space, 3.1 Send it to the target database; 3.2 Compute the sum of the similarities scores of the top N retrieval documents as the coherence score of the sequence.
4 Select the sequence (or sequences) with the best coherence score.
The Corpus1 method computes the coherence score for every path in the hypothesis space. This can be computationally expensive when the query terms have many possible translations. In our experiments, we reduced the hypothesis space by using a maximum of three translations for each query term. In cases where there were more than three alternative translations, we chose the three terms with the smallest distribution scores in the target corpus. For summing similarity scores, we set N to 100. 3.2.2 Corpus2 The Corpus2 method makes use of the mutual information of two terms based on corpus statistics. The method works as follows: 1 Get translations for each term in the source language query. 2 Construct a hypothesis space of translated sequences (overlapping n-grams, n=3 in our experiments) by obtaining all possible combinations of the translations in a source sequence of n query words. 3 For each translated sequence in the hypothesis space, 3.1 Compute mutual information (MI) scores for all term pairs in the sequence; 3.2 Sum up the scores from step 3.1 to give a coherence score for the sequence; 3.3 Select as a translation of the first source word in the sequence the alternative that gives the best coherence score.
4 Collate the selected translations into a new target language query.
The mutual information between two term t1 and t2 is defined as:
MI (t1, t2 ) = log
p(t1, t2 ) p(t1 ) p(t2 )
For Topic 136 (as presented in Section 2), we see that there are 3 × 10 × 1 × 3 or 90 possible ways the Spanish-toEnglish translations can be combined. For Chinese, the number is 1 × 2 × 3 × 4 × 5 or 120. Here, we give the target translations for the source query terms as determined by the above three translation disambiguation methods. (This reflects a “combined” method in which all the “best” terms of each method are retained for the final translated query.) In cases where the methods produced different “best” translations, we separate each candidate translation by a comma (“,”).
English: Leaning Tower of Pisa. What is the state of health of the Leaning Tower of Pisa? English terms: lean tower; pisa; lean; tower; health; state.
Spanish: Torre inclinada de Pisa . ¿En qué estado se encuentra la torre inclinada de Pisa? English translations of the Spanish query terms: pisa; high, tower; state; droop, tilt, bow.
English translations of the Chinese query terms: pisa; ter, tower; slant; health; affair, situation; health situation For our actual submissions reflecting a particular method (e.g., Web), we naturally used only those “best” terms that the method itself nominated. In addition to our official runs, our experiments included a combined method that uses the full set of terms (as given above) for the final target query. We describe the performance results of the combined method along with our official and baseline results in the next section.
Our CLIR experiment labels have the following convention: Cl<source-language>2<target-language><method>. Thus, “Cles2enw” denotes our Spanish-to-English Web-method run. The actual experiments involved separate runs for each of the three translation disambiguation methods described in Section 3 (the “w”, “t1”, and “t2” runs), and also runs with combined methods (“c1”, based on a combination of the Web and Corpus1 methods, and “c2”, based on all three methods). In all cases, combinations involved a simple concatenation of the selected translations nominated by each participating method. To establish a baseline for evaluating the quality of translation disambiguation, we used all possible translations in a default run (“all”). We ran English monolingual experiments to obtain the baseline with ideal translations.
All the experiments were run with post-translation pseudo-relevance feedback, as we have observed that
posttranslation pseudo-relevance feedback produces the best overall performance boost
For Chinese-to-English cross-language retrieval, compared with the bilingual baseline of keeping all possible translations, both the Web-based method and the Corpus2 method improve average precision and exact precision, while only the Corpus2 method improves recall. The Corpus1 method did not perform well compared with the baseline. Again, when the methods are combined, we observe improvements over the overall recall, average precision, and exact precision. The best run, with all three translation disambiguation methods combined, reached 89.0%, 59.7%, and 62.9% of the recall, average precision, and exact precision, respectively, of the English monolingual run.
Since pseudo-relevance feedback can affect performance differently depending on the original query terms, in our follow-up experiments, we re-computed the results without using pseudo-relevance feedback to best estimate the quality of the selected translations against the baseline. Table 3 and Table 4 give the results without feedback for both language pairs.
Regardless of whether pseudo-relevance feedback is used or not, the Chinese–English retrieval is overall poor. Beside translation ambiguity, the bilingual translation lexicon is very noisy, with many wrong word choices, occasional misspellings, and interfering descriptive text. In addition, wrong word segmentation resulted from incomplete segmentation dictionary coverage and the greedy longest match segmentation algorithm. For Spanish–English cross-language retrieval, all three translation disambiguation methods outperform the baseline in terms of recall, average precision, and exact precision (except recall with the Corpus1 method). The 2 The statistics reported here are higher than the official evaluation statistics. In the official submission, we did not filter out the consecutive single Chinese characters (>3) for this run. 3 The statistics reported here are higher than the official evaluation statistics since we fixed a formatting bug in the official submission. combinations of the methods outperform any individual method participating in the combinations. For Chinese– English cross-language retrieval, both the Web method and the Corpus2 method improve average precision and exact precision, while the Corpus1 method performs less well with these measures compared with the baseline. All three methods resulted in a decrease in recall performance. The combinations generally outperform any individual method participating in the combinations, except for recall of the Clch2enc1-nf run.
A summary of our results focusing on average precision is given schematically in Figure 1. In general, our best results for Spanish-to-English CLIR are virtually indistinguishable from English monolingual performance. Our best results for Chinese-to-English CLIR, suffering from the effects of poor resources, are at about 60% of the monolingual baseline. 5
Comparing the English baseline run to the Spanish-to-English Web run (Cles2enw), we find that 16 of the translated Spanish-to-English queries actually give better results in terms of average precision than the English queries, 26 are worse (of which 12 are much worse, with less than half the average precision of the English queries). (See Figure 2.)
•
Different word choice, e.g.: o o o o
Population in the English version of Topic 95 (Conflict in Palestine) becomes town in the Spanish-to-English translation of poblacion.
Ski races in the English version of Topic 102 becomes ski competition in the translations. The English version of Topic 106 (European car industry) contains countermeasures, whereas the Spanish contains medidas de recuperacion, which is translated by the methods described above as recovery measures, which are more common words than countermeasures, which is usually found in political rather than business contexts in the CLEF documents.
In Topic 140 (Mobile phones), the English version contains perspectives and the Spanish version contains pesrpectiva, which can be translated as outlook, perspective, prospect, and vista via our dictionary, and our method chooses outlook, a more common word, which might well account for why this topic scores better after translation. • • • •
1.2 n 1.0 o iis 0.8 rec 0.6 eP 0.4 rag 0.2 e v 0.0 inA-0.2 ce -0.4 en-0.6 r ff -0.8 e i D-1.0 o o o o o o o Different word ordering: Clarit recognizes noun phrases in the English text; in the Spanish-to-English text, a query is repeated backwards and forwards so that different phrases may be recognized in the reconstituted topic, e.g.:
Weapon destruction appears in the translation of Topic 119, whereas weapon and destruction are not found in the same simple noun phrase in the English topic.
Grunge rock appears in the translation of Topic 130 while the English contains grunge group. Different formulations of the same topic in English and Spanish, e.g.:
Topic 121 (Successes of Ayrton Senna) contains success and sporting achievements while the Spanish version contains just the word palmares, which is not in the Spanish-English dictionary and not translated in the resulting English version of the Spanish topic. It could be that the English words successes and achievements are only distractors from the relevant documents for Ayrton Senna.
Topic 138 (Foreign words in French) contains lengua, which translates to language not present in the English version.
98 211 91 301 138 95 001 114 140 120 102 115 131 119 106 125 134 133 112 99 316 103 111 108 113 105 116 128 137 97 261 92 212 104 123 135 124 94 071 139 109 129
Proper names written differently and not in the dictionary, e.g.:
Solzhenitsyn (Topic 94) is written Solzhenitsin in the Spanish topic and is retained with the same spelling in the translated queries since it is not found in the dictionary.
European Cup (Topic 113) is written Eurocopa in Spanish and is not in the dictionary, so it passes through as-is, but is not found as a string in the English documents.
In Topic 107 (Genetic Engineering), food chain appears as cadena alimentaria in Spanish, but alimentaria does not have food among its translations.
Dictionary divergences, e.g.: Comparing the English baseline run to the Chinese-to-English runs, we observe that most of the translated queries are worse in average precision compared with that of the English run. As in Spanish-to-English retrieval, the combination of the three methods (Clch2enc2) produces the best overall performance: 12 of the translated queries give better average precision than the English queries and 30 are worse (Figure 3).
Reduced ambiguity in translation, e.g.: • • o o 0.4 n iso 0.2 i c reP 0.0 e ag-0.2 r e vA-0.4 n ie-0.6 c n re-0.8 e f f iD-1.0 o o o o Difference in word choice, e.g.:
In Topic 113 (European Cup), the English version uses the word football, while the target translations include the word soccer. Since football can mean soccer or American football, the translation makes the query more relevant to the topic.
In Topic 133 (German Armed Forces Out-of-Area), the English version uses the word area, while the Chinese version uses border. As our system throws out out and of as stop words, the word area becomes too general, while border implies country boundaries.
138 113 133 95 151 114 135 119 131 108 122 92 301 140 128 98 99 261 137 97 361 91 061 124 111 109 100 94 071 134 121 112 105 120 116 139 103 104 123 129 102 125
•
Improper segmentation of words, e.g.:
In Topic 111 ( ), the word was segmented as two characters (with translations of act, arouse, get moving, move, stir, act, change, use, touch, etc.) and (with translations of draw, painting, picture). As a result, our system produced translations such as use picture instead of the correct translation animation.
Similar segmentation mistakes include: (galaxy) in Topic 129 as (silver) and (river); (beauty contest) in Topic 137 as (to choose, to elect, to pick, to select) and (America, beautiful, pretty).
(EU fishing) in Topic 139 was rendered as four single characters , , , , which were consequently filtered out by the system. •
Improper segmentation of transliterated names, e.g.:
) is segmented as: ; ; ; · ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; Topic 102 ( ; ; ; · ; The name · has been incorrectly segmented into seven single characters that have no direct semantic relation to the transliterated name. During pre-processing of the topics, we o o o filtered out the consecutive single characters. As a result, the query contained only general terms such as (victory), (ski), and (competition). The best precision was achieved by the Web run at 0.1078. In contrast, in the English query (Victories of Alberto Tomba), the name, “Alberto Tomba”, as a term makes the query very specific, resulting in high precision (0.7491). Other topics that suffered similarly include: Topic 94 (Return of Solzhenitsy), 98 (Films by the Kaurismäkis), 103 (Conflict of interests in Italy), 104 (Super G Gold medal), 120 (Edouard Balladur), 121 (Successes of Ayrton Senna), and 123 (Marriage Jackson-Presley). •
Different word choice, e.g.:
The translations of the word in Topic 107 ( hereditary, instead of the desired translation genetic.
In Topic 99, Holocaust is represented in Chinese as (butchery, massacre).
) are transmit, transmittal, and 6
We have explored three methods for selecting “best” target translations that take advantage of co-occurrence statistics among alternative translations of the source query words. We have demonstrated that, given a wide variety of possible translations that might be generated from a bilingual dictionary, the use of the Web or a local large corpus as a language model can provide a good basis for lexical choice, provided the gloss dictionary covers the source vocabulary. Combining the target words obtained from the different translation disambiguation methods can produce better cross-language retrieval performance, as compared to keeping all possible translations. For Spanish-to-English retrieval, our combined method achieved 95% and 97%, respectively, of the recall and average precision of our English monolingual run. For Chinese-to-English text retrieval, the recall and average precision reached 89% and 60%, respectively, of the English monolingual results.
Our experiments have shown that the quality of the translation resources has a significant impact on the performance of cross-language retrieval. The poor translation quality and poor coverage of the Chinese-toEnglish bilingual lexicon we used resulted in relatively poor performance of the Chinese-to-English retrieval. Names (transliterated names in CLEF topics, in particular) that are not covered in a translation lexicon need to be recognized and translated correctly for better retrieval performance.
With respect to the translation disambiguation methods, we believe our methods can be further improved by addressing the following issues: • • • •
Identifying the optimal context span for the Web method and the Corpus1 method. Currently, we use a 3word context, which may limit the access of contextual information.
Exploring ways to prune paths in the translation hypothesis space when the space is large. This includes the problem of how to rank translations of a query term when many translations are possible (currently, for the Corpus1 method, we use the idf scores of the translations from the target corpus) and how to rank the combinations of the translations of a sequence of source terms.
Identifying the optimal retrieval cutoff point for the Corpus1 method, instead of the arbitrary number of responses (i.e., 100) that we used in the experiments.
Incorporating phrasal translations. Phrasal translations can give important performance gain in crosslanguage retrieval. We have begun to explore techniques for automatically translating phrasal terms. In general, we feel our results demonstrate that it is possible to achieve remarkably high CLIR performance by exploiting relatively simple and available resources. Such approaches hold promise for cross-language retrieval in cases where machine translation, parallel corpora, or other knowledge resources may be difficult or impossible to obtain. We believe that incremental straightforward refinements of our approach will give better and more consistent results. We plan on testing this hypothesis in future work on other language pairs.