Thomson Legal and Regulatory participated in the monolingual, the bilingual and the multilingual tracks. Our monolingual runs added Swedish to the languages we had submitted in previous participations. Our bilingual and multilingual efforts used English as the query language. We experimented with dictionaries and similarity thesauri for the bilingual task, while we used machine translations in our multi-lingual runs. Our various merging strategies had limited success compared to a simple round robin.
2.1
Our participation at CLEF-2002 benefits from our earlier work, as well as from the work of others. Our bilingual effort relies on similarity thesauri for translating query terms from English to French or Spanish. In addition to translating words [6], we also translate word pairs which loosely capture noun and verb phrases. This differs from our approach last year when we generated word bigrams rather than pairs [3]. In addition, we follow Pirkola’s approach for handling multiple translations. By taking advantage of query structures available in INQUERY, Pirkola [4] has shown that grouping translations for a given term is a better technique than allowing all translations to contribute equally. This has been developed further by Sperer and Oard [7].
One of the main issues in multilingual retrieval remains collection merging. In our experiments, we use simple
merging techniques like round robin, normalized scores, as well as a variant of the CORI algorithm [
The WIN system is a full-text natural language search engine, and corresponds to TLR/West Group’s
implementation of the inference network retrieval model. While based on the same retrieval model as the INQUERY system
[
In addition, WIN has shifted from supporting mostly English content to supporting a large number of WesternEuropean languages as well. This was performed by localizing tokenization rules (mostly for French and Italian) and adopting morphological stemming. Stemming of non-English terms is performed using a third-party toolkit, the LinguistX platform commercialized by Inxight. A variant of the Porter stemmer is used for English. 2.2.1
WIN supports various strategies for computing term beliefs and scoring documents. We used a standard tf-idf for
computing term beliefs in all our runs. The document is scored by combining term beliefs using a different rule for
each query operator [
Query formulation identifies concepts in natural language text, and imposes a structure on these queries. In many cases, each term represents a concept, and a flat structure gives the same weight to all concepts. The processing of English queries eliminates stopwords and other noise phrases (such as “Find cases about”, or “Relevant documents will include”), identifies (legal) phrases based on a phrase dictionary and detects common misspellings.
In the experiments reported below, we use our standard English stopword and noise phrase lists, but do not identify phrases or misspellings. We have expanded the English noise phrase list with noise phrases extracted from queries used in previous years. Our German, French, Spanish, and Dutch runs use the same stopword lists as last year, but noise phrase patterns have been updated to cover query sets from CLEF-2001. Our Italian stopword and noise phrase list was validated by a native speaker, while our Swedish resources were extracted from the web and from available query sets.
Concept identification depends on text segmentation. In our experiments, we follow two main definitions for a concept: a concept is an indexing unit (typically a word) or a concept is a construct of indexing units. Constructs are expressed in terms of operators (average, proximity, synonym, etc.) and indexing units. For instance, we use a construct when a term has multiple translations, or when we identify word pairs. 3
Our approach for monolingual runs is similar to last year’s. We have revised the Italian stopword and noise phrase lists with the help of a native speaker. Our stemming procedure, although still based on the LinguistX toolkit, has been altered slightly to limit the occurrence of multiple stems.
German, Dutch, and Swedish are all compounding languages. However, the LinguistX platform does not support compound breaking for Swedish. We thus index and search using compound parts only German and Dutch content. Swedish is treated as a non-compounding language.
For all languages, we allow the stemmer to generate several stems for each term, as we do not rely on part-ofspeech tagging for disambiguation. Multiple stems were grouped under a single concept in the structure query.
Results from our official runs are reported in Table 1. All runs used the title and description fields from the topics. Our results are comparable to those of previous years. Introducing revised stopword and noise phrase lists for Italian allows us to achieve good performance.
While most languages achieve an average precision in the same range (between 0.4 and 0.5), the figures for Swedish are much lower. We suspect that not breaking compounds may be the main cause, since previous work with German and Dutch has shown that retrieval performance was enhanced by compound breaking. 4
Our bilingual runs were from English queries to Spanish and French collections. As in our previous work [3], we used a combination of similarity thesauri and machine-readable dictionaries. The machine-readable dictionaries were downloaded from the Internet (freedict).
We implemented a variant of the similarity thesaurus approach described in [6] for multilingual retrieval. We constructed two similarity thesauri: a word thesaurus and a word pair thesaurus. Both similarity thesauri were trained on a collection merging the UN parallel text corpus produced by the Linguistic Data Consortium, and an European Union (E.U.) parallel corpus that we have at TLR.
Using a part-of-speech tagger, we restricted the set of words to nouns, verbs, adjectives and adverbs. Word pairs were generated using sliding windows centered only on nouns, and components in pairs were ordered alphabetically. Terms, words, or pairs, were considered as translations when their similarity was above a predefined threshold. This threshold was chosen as the best configuration on CLEF-2001 data.
While we identified noise phrase patterns in our official runs, stopwords were expected to have a different part-of-speech (like auxiliary, prepositions, etc). We later added a stopword list in conjunction with noise phrase patterns.
Table 2 reports our official runs. The translation resources for our official runs were a combination of the word similarity thesaurus and the dictionary.
Table 3 summarizes our unofficial runs. These runs used an explicit stopword list, instead of relying on part-ofspeech tags. We also translated word pairs after we completed training the word pairs similarity thesauri. The last runs use automatic translation and are part of our multilingual run.
Stopwords Stopwords + Pairs Machine Translation
Stopwords Stopwords + Pairs Machine Translation
English/Spanish English/Spanish English/Spanish English/French English/French English/French
A comparison of Tables 2 and 3 shows that using an explicit list of stopwords helps enhance the average precision. We have identified inaccuracies in part-of-speech tagging as one of the main reasons. Inaccuracies are often caused by inadequate context, or by the lack of a specific tag in one of the languages, e.g. auxiliary versus verb.
Our approach using similarity thesauri has some shortcomings in comparison with the machine translation approach. In particular, it is very dependent on the parallel corpus used for training. In our experiments, using E.U. material lead to some E.U.-oriented translations. For instance, European is translated into the French terms europe´en and communaute´, and the Spanish terms europeo, comunidad and constitutivo. One way of addressing that issue may be to filter out corpus-specific terminology.
Unlike our results with bigrams at CLEF-2001 [3], translating word pairs provides little advantage over translating individual words. One plausible hypothesis is that the window used to generate word pairs (we used a window of 9 centered on a noun) and the query structure are not compatible (we used a phrase node, i.e. a proximity of 3). 5
During our multilingual experiments, we translated queries only. We used the indices generated for the monolingual runs for German, French, English, Italian and Spanish. Queries were translated from English to the other languages using Babelfish.
Our main focus was merging, although we have not been very successful so far. We tried a variety of merging
approaches:
• round robin, i.e. a rank-based approach that alternates documents from each collection. In our setting,
documents with identical score were given the same rank.
• raw score, which may or may not be comparable across collections
• CORI, where the collection score is estimated by the maximum score a (translated) query can achieve on
that collection, not the original collection score in Callan et al [
cori score = score within collectioni ∗ (1 + nb lang ∗ collection scorei − avg collection score ) avg collection score • normalized score, where the maximum score within collectioni is the score of the document at rank 1. norm score =
score within collectioni − 0.4 maximum score within collectioni − 0.4 0.4 represents the minimum score any document can achieve in the belief network retrieval model. • collection-weighted normalized score, where collection scorei is the same as in the CORI approach above, and maximum collection score is the maximum of these scores.
collection scorei weighted norm score = norm score ∗ maximum collection score
Our official run tlren2multi used round robin. Table 4 reports results from the different merging approaches. As reported too often, we found it hard to outperform the round robin approach. Our collection-weighted normalized score is the only merging approach to perform better but the difference is not significant. Our results with the CORI merging strategy are comparable to those obtained by Savoy [5]. It is possible that the CORI algorithm is impacted by our choice of collection scorei. More analysis is required to assess the difference between the original CORI and our version.
Run ID tlren2multi (round robin) raw score cori score norm score weighted norm score
0.2049 0.1883 0.1023 0.1827 0.2160
0.2803 0.2521 0.1489 0.2496 0.2794
There are two issues with multilingual retrieval, the quality of the individual runs and the effectiveness of the merging strategy. The quality of the individual runs can easily be assessed by comparing their performance to the performance of monolingual runs. As can be seen in Table 5, using translated queries leads to an average degradation of 25% in performance (performance is measured in terms of average precision).
How to quantify the effectiveness of merging strategies remains an open issue. We can observe the following properties in an attempt to measure the effectiveness of merging. In Table 6, we observe that merging better
French Spanish Italian individual runs (the monolingual column vs. the translated column) leads to better performance. We can also compare the average of the individual run performances with the performance of the multilingual runs, and find that the average of individual runs is higher that any multilingual run. These observations tend to indicate that merging also deteriorates the effectiveness of multilingual runs, but do not tell us how much so.
cori score norm score weighted norm score Average of individual runs
The poor performance of our English monolingual run (around 25% average precision) 1 had a noticeable impact on multilingual runs. We found that round robin, cori score and weighted norm score were not affected as much as raw score and norm score by the English run. We expected round robin to be more sensitive to English documents, since one fifth of the documents are English. In effect, our modified version of round robin limited that effect for 40 queries, and aggravated it for 10 others. As could be expected, raw score was mislead by the higher score of English documents for a large number of queries. norm score suffers a similar problem: it is mislead when document scores are close to the highest document score in the retrieved list. 6
Our participation at CLEF-2002 has mixed results. On the one hand, we consider that our monolingual runs successful, even though we intend to evaluate how much improvement can be achieved by relevance feedback. On the other hand, our bilingual and multilingual runs did not lead to the expected results. For instance, we did not find any evidence that translating word pairs was helpful in our bilingual runs. We also encountered an over-fitting problem when training similarity thesauri on the E.U. corpus. Finally, we are still in the process of investigating alternative merging algorithms, since our current approach has shown limited success.
1We suspect that this run encountered problems. We have not yet identified what the issues are. [3] H. Molina-Salgado, I. Moulinier, M. Knutson, E. Lund, and K. Sekhon. Thomson legal and regulatory at clef 2001: monolingual and bilingual experiments. In Workshop Notes for the CLEF 2001 Workshop, Darmstadt, Germany, 2001. [4] A. Pirkola. The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval.
In Proceedings of the 21th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 55–63, Melbourne, Australia, 1998. [5] J. Savoy. Report on clef-2001 experiments. In Workshop Notes for the CLEF 2001 Workshop, Darmstadt, Germany, 2001. [6] P. Sheridan, M. Braschler, and P. Schuble. Cross-lingual information retrieval in a multilingual legal domain. In Proceedings of the First European Conference on Research and Advanced Technology for Digital Libraries, pages 253–268, Pisa, Italy, 1997. [7] R. Sperer and D. W. Oard. Structured translation for cross-language information retrieval. In Proceedings of the 23th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 120–127, Athens, Greece, 2000. [8] H. Turtle. Natural language vs. boolean query evaluation: a comparison of retrieval performance. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 212–220, Dublin, Ireland, 1994.