This paper describes the official runs of TNO TPD for CLEF-2001. We participated in the monolingual, bilingual and multilingual tasks. The main contribution of this paper is a systematic comparison of three types of translation resources for bilingual retrieval based on query translation. We compared several techniques based on machine readable dictionaries, statistical dictionaries generated from parallel corpora with a baseline of the Babelfish MT service, which is available on the web. The study showed that the topic set is too small to draw reliable conclusions. All three methods have the potential to reach about 90% of the monolingual baseline performance, but the effectiveness is not consistent across language pairs and topic collections. Because each of the individual methods are quite sensitive to missing translations, we tested a combination approach, which yielded consistent improvements up to 98% of the monolingual baseline.
(1) 1. The formalisation is ad-hoc. 2. It is hard to normalise scores.
In previous years (CLEF2000) we have seen that adding a so-called document prior conditioned on the document length results in a significant improvement of retrieval performance especially for short queries. The document prior was simply integrated into the model as a multiplication, assuming independence.
The approach has some problems: 3. Its effect on short (title) queries is much larger than on longer queries.
Especially problem (2) is important for CLIR, because for the multilingual task, we want to normalise scores. Document scores in the original model are linearly related to the query length. Now suppose, we translate the query into different languages using Systran. It could very well be the case that a translation into a compounding language results in a shorter query, thus the translated queries will not have a homogeneous length. An easy strategy is to divide the document scores by the query length. But this division would also affect the “document prior”. We propose to model the a-priori probability of relevance conditioned on the document length and the normalised generative probability of the query based on the document unigram model as separate models. P P (T ability measure (Ti) defines the probability of drawing a term at random from the collection, ijDk) P probability of relevance (Dk) is usually taken to be a linear function of the document length, modelling i defines the probability of drawing a term at random from the document; and defines the importance ; query length), which sample space is the set ft(0); t(1); t(m)g of all terms in the collection. The probi i n formula, Ti is a random variable for the query term on position in the query (1 n, where is the Formula 1 shows the basic idea of this approach to information retrieval, where the document-based language model is interpolated with a background language model to compensate for sparseness. In the of each query term. The optimal (0.3) was found by tuning on earlier CLIR collections. The a-priori the empirical fact that longer documents have a higher probability of relevance.
The retrieval model has been extended for the CLIR task, by integrating a statistical translation step
into the model [
An informal paraphrase of the extension is: the relevance of a document in a target language with respect to a query in a different source language can be modelled by the probability that the document query term Si, we also assume for the moment that we know their respective translation probabilities. The calculation of the probability involves an extra step: the probability of generating a certain query term is the sum of the probabilities that a document in the target language generates a word which in turn is translated aspect of the approach is that the model treats alternative translations as a probabilistic disjunction resulting in highly structured queries. (2)
P We estimated the “reverse” translation probabilities (SijTj) (the translation probability of a term in the
In this paper we will compare these three types of resources in a quantitative and qualitative way. For
the machine readable dictionaries we used the VLIS lexical database of lexicon publisher Van Dale. For
a description of the structure and contents of the VLIS database we refer to our CLEF 2000 paper [
The major part of the experiments was based on an indexing procedure which involves morphological
lemmatisation. The lemmatiser is based on the Xelda morphological toolkit 1, developed by XRCE
Grenoble, which supports a wide range of European languages. The lemmatiser removes inflectional suffixes,
performs case normalisation (proper nouns and German nouns keep their capitalisation) and can segment
compounds. This year, we tested whether keeping the capitalisation would improve precision (cf. Section
6.1). For German and Dutch, the basic procedure for handling compounds was to add both the full
compound (inflection removed) and the recognised compound segments to the set of index terms (at indexing
and retrieval time). This approach has proved to work well [
P (T ment normalisation which comes with the maximum likelihood estimates for ijDk). However, there = of . For our CLEF experiments, we used formula (3) with 0:2, which was found to give optimal
The scores of these models are subsequently interpolated via linear interpolation, in formula (3) the final retrieval status value (RSV) is composed of two components, the first addend is the a-priori probability of relevance of a document conditioned on its length, the second component is the unigram component.
In studies on other collections we found that the optimal value for the interpolation parameter is
relatively stable across collections, but is dependent on the query characteristics. The optimal value of the
interpolation parameter is inversely related to the query length. This can be explained as follows: for
short queries, the probability of relevance is not dependent on the document length because of the
docuis empirical evidence that longer documents have a higher probability to be relevant [
1cf. http://www.xrce.xerox.com/ats/xelda/overview.html While the development of a theoretical framework for CLIR is extremely important, a more practical but not less important issue is the acquisition or development of translation resources. CLEF 2000 showed that there are several classes or resources can be used for successful CLIR.
1. High quality machine readable dictionaries are available for the major European languages[
CLEF 2000 showed successful use of the Babelfish translation service based on Systran, the
Powertranslator system from L&H and several other systems[
/department/tt/english/home.html and /department/tt/italian.html. 4. (For each pair) download web pages, perform a language check using a probabilistic language clas
sifier, remove pages which are not positively identified as being written in a particular language.
We contrasted query translation using VLIS with translation based on the Babelfish MT service and word by word translation based on dictionaries derived from a collection of parallel web documents.
For CLEF 2000 we had already developed three parallel corpora based on web pages in close
cooperation with RALI, Universite´ de Montre´al. We used the PTMiner tool [
The parallel corpora were used to train simple translation models (IBM model 1, [
The mining process was repeated for three language pairs (EN-IT, EN-DE and EN-NL) and resulted in three modestly sized parallel corpora. RALI had already mined a parallel web corpus for English-French. Table 1 lists some quantitative data about these corpora.
nr of candidate pages 1053649 1828906 1170082 n.a.
nr of candidate pairs 23447 33577 24738
n.a. 1. Query a web search engine for web pages with a hyperlink anchor text “English version” and respective variants. 6.1
Our main goal for the monolingual runs was to improve the pool and to provide a baseline for bilingual experiments. We also did a minor experiment with case sensitivity and fuzzy query term expansion.
description
Table 2 shows the results of our monolingual runs. We added some unofficial runs to complete the picture. Since case insensitivity had deteriorated the performance of some topics in previous years (when a proper name and a normal noun are homonyms e.g. Turkey and turkey) we decided to test the proper name recognition function of the Xelda morphological analyser. We did not apply any case normalisation after the Xelda lemmatisation step. Xelda can often correctly disambiguate homonyms, but when the context is not sufficient, our indexer chose the proper noun reading.
As in previous years, we also tested the effectiveness of a fuzzy lookup scheme. If a query term does not match with a term in the index vocabulary, a fuzzy lookup procedure (based on n-grams) substitutes the nearest matching term. This function is practical for misspellings in queries. The same fuzzy matching procedure can also be used to expand the query with near matches of all query terms. This function can be practical for queries that contain diacritics or such as accents, or spelling variants like the German ß/ss. The CLEF 2001 monolingual runs showed some modest improvements for the fuzzy expand option. We suspect however, that this is mostly due to undoing the case sensitivity, which is a side-effect of the expansion operation. For most languages, fuzzy conflation increases average precision by about 1% (absolute scale), the only exception being French. We will look at two languages (French and Dutch) in some more detail.
For French, mean average precision is hurt by the expansion operation. Looking closer at runs tnoff1 and tnoff3 shows that the main differences occur at topic 70 an 75. Topic 70 is about Kim Il Sung while Le Monde almost invariably spells his name as Kim Il-sung. The tokeniser converts dashes into spaces, a strategy which is necessary here, but it is clear also that case normalisation is necessary for proper matches. Omitting case normalisation (tnoff1) is 0.14 % less effective than doing case normalisation (tnoff3). Topic 75 is an odd topic because it only contains one relevant document; run tnoff3 has the relevant document on the first position, whereas run tnoff1 retrieves the document in third position. The key point here is that the topic mentions tuerie. The baseline run retrieves the single relevant document in first position, the fuzzy expansion run conflates tuerie with tuer which has the effect that two other marginally relevant documents have an increased score and the single relevant document has rank 3, decreasing the map for this topic with 0.666. We did a control run (tnoff4) which was similar to the baseline (no fuzzy expansion) but which did apply case normalisation. The run performed marginally better at 0.4904, confirming our conjecture that our sub-optimal proper noun recognition module effectively hurts average precision.
For Italian, we also have conclusive evidence that case sensitivity deteriorated retrieval performance slightly instead of an expected improvement. There are marked differences only for a small number of topics. E.g. topic 88 talks about Spongiforme Encefalopatia. The Italian Xelda lexicon does not contain these words, and thus the capitalisation is left in place, which hurts performance here, because in most documents these terms are used in lowercase. A possible strategy could be to lowercase unknown words, but this information is not available for all languages.
For Dutch the fuzzy expansion run increases mean average precision w.r.t the baseline run. The increase is mainly due to topic 52 (0.0165 versus 0.5000). Here, the effect is merely due to case normalisation: Xelda sometimes assigns the lemma Chinese and sometimes chinese, which causes a great loss of recall in the baseline (case sensitive) run. The control run (tnonn6) confirms again that case sensitivity hurts average precision in the current set-up.
Our conclusion is that in principle it is good to do proper noun recognition (which we implemented by capitalisation) because sometimes a proper noun is homonymous with another word (e.g. in English: (Kim Il) Sung and sung). However, if the proper noun recognition is not perfect, e.g. it cannot deal with spelling variation (Kim Il-sung versus Kim Il Sung) it might hurt recall more than it improves precision. In practice, an imperfect proper name recognition module will yield inconclusive results which depend on the specific topic set. Our conjecture is that a proper name recognition module in combination with a proper name normalisation module (which conflates spelling variants of proper nouns) could improve retrieval effectiveness in a convincing way. The fuzzy matching technique as a means to find spelling variants other than case alternatives can help to improve retrieval performance as a fall-back option (i.e. when a query term is not found in the indexing vocabulary). Expanding every query term to find spelling variants is probably only useful when it is constrained to special word classes: 2. spelling variants of proper nouns: transliteration of non-Latin script names into a Latin script is usually done in different ways.
A principled approach would be to recognise these word classes (which are language dependent!) and build tailored normalisation/conflation modules for each of them. Our non-principled fuzzy conflation procedure does the job, but often also hurts precision because it is not constrained to these word classes.
The second experiment was a comparison between Xelda inflectional lemmatisation and a Dutch variant
of the Porter stemming algorithm [
The 4th column of Table 2 shows the percentage of documents of the top 100 which is judged. In comparison with last year, this percentage has increased slightly, possibly due to the increased pool depth (60 instead of 50) and/or the increased number of participants. Only the English pool is of a slightly lower quality, questioning again why monolingual English runs are excluded from the pool. 6.2
FR-EN: Babelfish versus corpus A striking result is that the web corpus runs perform at the same level as the Systran based Babelfish service. Again we looked at some topics with marked differences in average precision. First, the topics where the web corpus run performs better: in topic 47 (+0.55), Systran lacks the translation of Tche´tche´nie (Chechnya); topic 58 (+0.46), Systran translates mort and mourir with died and die, whereas the web corpus has the additional concepts of death and dead; topic 82 (about IRA attacks, +0.4) Systran translates l’IRA erroneously by WILL GO, the corpus based translation brings in the related term bomb as a translation of attack. Secondly, the topics where Systran performs much better: topic 65 (-0.39) the corpus translations of tr e´sor are treasury and board, which would be a fine phrase translation. In this context however, tr e´sor does not have the financial meaning and because our system does not recognise phrases, treasury and board are used as separate query terms, which has the effect that the much more frequent term board, brings in a lot of irrelevant documents. Topic 75 (-0.98) suffers from a wrong interpretation of the word sept, which is translated to sept (September) and 7, the latter term is discarded by the indexer. The month abbreviation retrieves a lot of irrelevant documents, resulting in a low position of the single relevant document; in topic 80 (about hunger strikes) faim is translated both by hunger and death. Death might be a related term in some cases, but it also retrieves documents about strikes and death, hurting precision; topic 89 talks about an agent immobilier, Systran produces the correct translation real estate agent, but the corpus based translation has officer and agent as additional translation. Here, the phrase translation of Systran is clearly superior.
run tag tnoee1 tnoee3 tnofe1 language pair EN-EN EN-EN
FR-EN tnofe1a
FR-EN tnofe2 tnofe3 tnonn1 tnoen1 tnoff1 tnoef3 tnoef4 tnoef5 tnoii1 tnoei3 tnoei4 tnoei5
FR-EN FR-EN NL-NL EN-NL FR-FR EN-FR EN-FR EN-FR IT-IT EN-IT EN-IT EN-IT FR-EN: Babelfish versus VLIS We also looked at some topics that revealed marked differences between the Systran run and the VLIS run. Topic 58 is a clear example where VLIS gives the best results (+0.44) , it correctly translates the key term euthanasie by euthanasia instead of the non standard translation euthanasy by Systran. In most cases however, Systran gives better results, some examples: topic 79 (-1.00), here VLIS fails to translate Ulysse into Ulysses, the word by word translation strategy also fails for sonde spatiale, VLIS translates sonde into sampler;sound;probe;catheter;gauge;plumb;sink;auger and spatiale into spatial;dimensional. Probably the fact that the query terms Ulysses and space are missing is more detrimental then the fact that VLIS generates some irrelevant translations for sonde, since the correct translation (probe ) is found. In topic 62 (-0.50) both Japon is not found in VLIS and the multi-word unit tremblement de terre is not recognised as the French translation of earthquake. In topic 66 (-0.50) the seminal proper noun Lettonie is not found in VLIS, but is successfully translated by Systran. The proper nouns are probably not found in VLIS because in French, country names are usually denoted in combination with a determiner La France, Le Qu e`bec,..., our lexical lookup routine was not aware of this fact. In topic 80 (-0.65) the seminal query term faim is translated to appetite;lust instead of hunger (Systran). In topic 83 (-0.40), VLIS translates ench e`re by raise;bid, whereas Systran gives the contextual better translation auction. Finally, the low effectiveness of the VLIS based translation for topic 86 (-0.50) is due to a combination of factors, the dictionary based translation is rather fertile (e.g usage is translated in currency;commonness;use;custom;habit;way;practice;usage;word) and also the fuzzy expansion process is active, to correct for the case sensitivity of the index, which brings in more unwanted terms. Summarising, the Systran based Babelfish service outperforms the VLIS based run, because i) VLIS lacks translations of some proper nouns, ii) the word by word based translation fails for some topics (we currently have not accessed the phrasal translations in VLIS) and iii) VLIS has no method for sense disambiguation. Babelfish most probably uses phrase translations as a form of contextual sense disambiguation: the translation in isolation of enche`res is bidding, ventes aux ench e`res gives auction sales and ventes ench e`res gives sales biddings.
EN-IT and EN-FR We do not want to base our judgement of the effectiveness of translation resources on one language pair and one topicset. Therefore we included two other languages pairs: EN-IT and ENFR. For these language pairs, the VLIS based runs is clearly superior, which is not trivial, since these translations use Dutch as a pivot language. The EN-IT web based run performs surprisingly good (better than Systran), given its relatively small size and the fact that the corpus is hardly cleaned. CLEF 2000 topic collection For an even better perspective, Table 4 shows the results for the same language pairs based on the CLEF 2000 topics.
method mono web corpus Babelfish VLIS mono web corpus Babelfish VLIS mono web corpus Babelfish VLIS language pair FR-FR EN-FR EN-FR EN-FR EN-EN FR-EN FR-EN FR-EN IT-IT EN-IT EN-IT EN-IT
When we make a comparison of CLEF 2000 and CLEF 2001 results, we hardly see any consistent results, this confirms experiences we had with the various CLIR tracks at TREC6/7/8. The bilingual results depend strongly on lexical coverage. When a resource misses a few important concepts in the topic collection, its performance is seriously affected. In other words the mean average precision of a run is proportional to the lexical coverage. Unfortunately a set of 50 topics proves to be too small to measure the retrieval performance of a system based on a particular translation resource and its inherent lexical coverage in a reliabel way. We could do a few things to remedy this problem:
Devise a special test for lexical coverage.
Remove topics from the collection, for which one of the methods has serious lexical gaps. This
might very well introduce a bias, but has the advantage that we can concentrate on some interesting
research questions:
– How well do the different methods deal with the translation of phrases?
– Is query term disambiguation really necessary?
– Can we exploit synonym translations?
Combination of translation resources If we are merely interested in a strategy yielding “the best” result,
it is fairly obvious that a combination of lexical resources could help to remedy the gaps of the
individual translation resources. Some groups experimented with this idea, but with mixed results[
Table 6 presents the results of our official multilingual runs. We can make several observations, the VLIS based run performs a little bit better than the Babelfish based run based on English topics. This comes as a surprise after the detailed comparison of Babelfish and VLIS on the bilingual FR-EN task. The only explanation could be that the translation quality of the various language pairs offered by Babelfish and/or Systran is not homogeneous. We will assess this shortly. Secondly, the VLIS run based on Dutch topics performs better than the VLIS run based on English topics. This is not so obvious, since the NL-X tnoex3 tnoex4 tnonx3 language pair mean average precision EN-X EN-X NL-X run does not contain a monolingual EN-EN run. Apparently the advantage to use the direct translations instead of translating from English via Dutch outweighs the disadvantage of replacing a monolingual (ENEN) by a bilingual (NL-EN) run.
We have computed the mean average precision of the partial runs which make up the multilingual runs, results are shown in table 7. The number of topics on which the mean average precision is based is shown between brackets. We discovered that something went wrong with the Babelfish translations for some of the topics from English into Spanish (hence the 47 instead of 49).
Rerunning the Babelfish EN-X multilingual run yielded a mean average precision of 0.2465, which does not really change the picture.
When we compare the Babelfish bilingual EN-X translation runs with the official FR-EN run (90%), we see that these runs compare less favourable with respect to the corresponding monolingual run: ENFR:86%, EN-DE: 72%, EN-IT: 64%, EN-ES: 80%. Indeed, the translation quality of Babelfish is not homogeneous across languages for this set of topics. 7
At CLEF 2001, we concentrated on monolingual and bilingual experiments. Our hypothesis was that proper noun recognition could improve precision, because proper nouns are sometimes homonymous with other words. Our implementation of a proper noun aware indexing strategy turned out to hurt average precision. This is probably caused by a sub-optimal way to deal with the sometimes sill ambiguous output of the Xelda lemmatiser. We also experimented with a fuzzy query expansion method, in other to deal with spelling variation of especially proper nouns. A control experiment showed that the effectiveness of this algorithm is largely due to the conflation of capitalised/uncapitalised forms. We suggest a class based expansion scheme instead. Further we compared two different lemmatisation schemes for Dutch: the morphological lemmatisation (which includes the decomposition of compounds) proved to be markedly more effective than the Dutch variant of the Porter suffix stripper. For the bilingual task, we compared three different translation resources: a bilingual MRD (VLIS), a statistical dictionary based on a parallel web corpus and the Babelfish MT service. For the translation pair French-English, the web based and the MT based run reach a quite impressive level of 90 and 92% of the monolingual EN-EN run. The VLIS based run reached a level of 73%, which is due to several factors: deficiencies in the lexical lookup of proper names, lack of phrase handling and translation via a pivot language. For the translation pair EnglishDutch, the VLIS based run scored better at 87% of the monolingual baseline, but failure analysis showed that phrase translation could improve results substantially. We think that the good results of the Babelfish based runs are mostly due to its ability to translate phrases. We consider the competitive results of the runs based on a web corpus based dictionary as a breakthrough in CLIR, because parallel web corpora for EN-* language pairs are relatively easy to acquire. We hope to improve upon these results by training more complex models which allow for phrase translations. We also looked at several other bilingual tasks and did a comparison with CLEF 2000 topics. Our conclusion is that the topic sets are too small to really compare techniques to integrate translation resources into the retrieval model. Retrieval performance is proportional to lexical coverage. The set of 50 topics is too small to estimate lexical coverage, thus results are highly dependent on the particular topic set. We tested a combination approach, which merely concatenates the translated queries and proved to be consistently effective. Finally, in the multilingual task, our best result was achieved by a run based on the Dutch topicset and the VLIS lexical database.
We thank XRCE Grenoble for making the Xelda morphological toolkit available to us. We thank Van Dale Data to make an extended version of VLIS available which includes translations into Italian. Furthermore we would like to thank Michel Simard (RALI, Universite´ de Montre´al ) for helping with the construction of aligned corpora and building translation models. We also thank George Foster and Jian-Yun Nie (also RALI) for general discussions about the application of statistical translation models for CLIR.