Thomson Legal and Regulatory participated in the monolingual track for all five languages and in the bilingual track with Spanish-English runs. Our monolingual runs for Dutch, Spanish and Italian use settings and rules derived from our runs in French and German last year. Our bilingual runs compared merging strategies for query translation resources. Thomson Legal and Regulatory (TLR) participated in CLEF-2001 with two goals: reuse of rules and settings inside a family of languages for monolingual retrieval, and start our effort on bilingual retrieval. In our monolingual runs, we considered Dutch and German as being one family of languages, while French, Spanish and Italian formed another. We used the parameters we derived from our runs at CLEF2000 for German and French for each language in their respective family. In addition, we investigated the use of phrases for French and Spanish document retrieval. Our first attempt at the bilingual track was from Spanish queries to English documents. In that task, we experimented with combining various resources for query translation. Our submitted runs used similarity thesauri and a machine-readable dictionary to translate a Spanish query into a single English query. We also compared our official runs with the merging of individual runs, one per translation resource. In this paper, we briefly present our search engine and the settings common to all experiments. Then, we discuss our bilingual effort. Finally, we describe our participation in the monolingual track.
usually relies on various resources: a stopword list, a list of noise phrases (“Find cases about…”, “A relevant document describes…”) , a dictionary of (legal) phrases, and a list of common misspelled terms. We used stopword and noise phrases lists for all languages, while for French and monolingual Spanish, we also used a phrase dictionary. We used our French and German stopword lists from last year, the Dutch list given on the CLEF homepage, and compiled Spanish and Italian stopword lists from various sources on the Web. For all languages, we extracted introductory phrases from the query sets of previous CLEF and TREC conferences. As we had no Italian speaker in our team, our introductory list in Italian is very limited and simple.
Finally, we submitted two sets of runs: runs including only the title and description fields from the CLEF topic, and runs including the whole topic. The former runs are labeled with ‘td’ and doubled weighted the title fields. The latter are labeled with ‘tdn’ and used a weight of 4 for the title field, 2 for the description field, and 1 for the narrative.
In our bilingual runs, we concentrated on query translation and more specifically the combination of various translation resources. We used three main resources, a machine-readable dictionary (MRD) that we downloaded from http://www.freedict.com and two different similarity thesauri. Coverage of these resources is reported in Table 1.
We implemented a variant of the similarity thesaurus approach described in [PBS97] for multilingual retrieval. We used a parallel corpus, the UN parallel text corpus produced by the Linguistic Data Consortium. We generated two different thesauri: a unigram thesaurus and a bigram thesaurus. Our intent with the bigram thesaurus was to capture some phrase translation. We limited the number of bigrams by constraining bigrams to not contain stopwords, and by frequency thresholding. We used at most 15 translations from each thesaurus, and also used a threshold on the similarity to filter out translations that we thought would not be helpful. This threshold was determined on training data from CLEF 2000. We used all translations from the MRD. In all cases, multiple translations of the same Spanish term were grouped as the same concept given a translation source. We investigated two main approaches to combine our translation resources: a priori merging, i.e. combining translations during query construction, and a posteriori merging, i.e. merging runs produced by queries translated from a single resource. For a posteriori merging, we used a score-based and a rankbased technique to generate the merged score. The score based technique relies on a feature of the WIN engine. WIN computes the best score a document can achieve for a given query. We used that maximum score to normalize individual runs. Normalized runs are merged in a straightforward manner. The rank based technique is also fairly simple. The score in the merged result list is a function of the ranks in the original lists. Here, we report experiments using the sum of the logarithms of the document rank in each run.
Our official runs relied on the a priori approach, combining translations during construction. Runs tlres2entdw and tlres2entdnw2 combined only the unigram thesaurus to the dictionary, while runs tlres2entdb and tlres2entdb combined both thesauri with the dictionary. In 2 The only difference between runs tlrdetdw and tlrdetdnw and between runs tlres2entdb and tlres2entdnb is whether the narrative field is used or not. Official runs
Unofficial runs 0.3846 Comparison to the median is indicative. If these runs had been included, the median will be different.
The poor performance of the bigram thesaurus also impacted our a posteriori merging. Indeed, runs using the bigram thesaurus show lower average precision than runs only using the unigram thesaurus and the MRD. The score-based technique performed better than the rank-based technique. The rank-based technique reported here is based on the product of the ranks. As a result, documents with very different ranks in the individual runs are penalized.
Finally, a priori merging performed better than both techniques for a posteriori merging. One reason is that the number of non-translated terms diminished when resources are combined a priori. Further analysis is needed to better understand the difference in behaviors.
In our monolingual runs, we considered two aspects: families of languages, and the use of a phrase dictionary. We used the same rules for Dutch and German on the one hand, and French, Spanish and Italian on the other. In addition, we introduce a phrase dictionary in some of our Spanish and French runs. German and Dutch were considered as compounding languages. Using the LinguistX morphological analyzer allowed us to detect compound words and break them at indexing and search time. We used a structured query and loose noun phrases to represent compounds.
For French, Spanish and Italian, we allowed the LinguistX morphological analyzer to generate several stems (we did not disambiguate using part-of-speech tags). Multiple stems were grouped as a single concept (using a OR/SYN or a SUM node for instance) in the structured query. For French and Spanish, we generated a dictionary of roughly 1000 noun phrases. We extracted noun phrases from the French and Spanish document collections, we then derived some rules to filter out proper nouns like “Bill Clinton” and phrases we thought non-content bearing such as “année dernière” or “premier trimestre”. Finally, we manually filtered the 1500 most frequent noun phrases to remove noisy phrases not captured by our simple rules. Examples of phrases are “unión europea” and “casque bleu”.
Using a phrase dictionary was neither harmful, nor helpful. We observed that phrases from the dictionary were found in only one fifth of the queries. For those queries, there is no clear emerging behavior: some perform better using phrases, while others do not. The difference in precision per query between the two runs is usually very small. Our results for compounding languages are in the better half of the participants for these runs, so are our Spanish results. We believe that reusing of settings in a family of languages is indeed helpful. We need to perform some further analysis to confirm that belief.
Our Italian run was hindered by the lack of a good noise phrase list, as some of our structured queries still contained terms like information or document.
While we used last year’s settings for French, we did not achieve the performance we were aiming for. So far, we have identified two reasons. First, our noise phrase list for French missed capturing some of the patterns used in this year’s topics. When we manually cleaned the topics, we observed an improvement in the average precision. Some topics, however, benefited from non-content bearing terms that were not very frequent in the collection (for instance énumérant in queries 59 and 71). Next, while we originally intended to consider a term with multiple stems as a single concept, we realized that our scoring was overweighing such a term. Changing the behaviour would also have helped our French runs.
One of the problems in our bilingual runs was the coverage of the translation resources. Many translated queries still included original Spanish terms. In order to solve that problem, we can either use a MRD with a wider coverage (20,000 entries is a rather limited dictionary), or try to get a better coverage from the similarity thesauri. Better coverage may be achieved by using a parallel/comparable corpus in the same domain as the retrieval collections, if not the retrieval collections themselves (see [SBS97]). We will be investigating alignments of documents in related collections in the future.
Our monolingual runs contain no query expansion or pseudo-relevance feedback. Once we have refined the list used in query processing, for instance adding a list for misspelled terms, we will focus on automatic query expansion to try and enhance our searches.