We participated in CLEF'2001 with four automated bilingual runs. UTACLIR is an automatic query translation and construction system for cross-language information retrieval. The system automatically extracts topical information from request sentences written in one of the source languages and constructs a target language query, based on translations given by a translation dictionary. The new features for the CLIR process from Finnish, Swedish and German to English focus on matching compound words and a new n-gram based technique for matching proper names and other non-translatable words. The results for all the four runs are good. Average precision for all the queries shows clear improvements. For German - English we have tested two types of dictionaries (two runs). The first one included all translations from the standard dictionary. The second contained the same data, except that all direct translations of compounds were excluded. The test with two dictionaries for the German runs gives an indication that the new features in the UTACLIR process work well also with a limited dictionary.
In the query construction phase the right use of windowing techniques and phrase construction is emphasised
This is true especially for CLIR-queries where compound splitting and translation of components is performed.
In our method for handling compounds we have experimented with the window size, and the phrase operator. In
last year CLEF-tests we used an operator requiring strict word order and a fairly small window size. This year
we allow for a free word order and a broader window size in the phrase construction for compounds.
Proper names often are prime keys in requests, and if not translated by dictionaries, query performance may be
ruined. However, the fact that proper names often are form variants of each other allows the use of approximate
string matching techniques to find the target language correspondents for the source language names.
Approximate matching techniques involve Soundex and Phonix, which compare words on the basis of their
phonetic similarity
We will present the new features of the UTACLIR research process in this year’s CLEF. The old process that is
used as a base in the development process is described in detail in the Working Notes and Proceedings of the last
year’s CLEF
Our approach to solve the general problems for bilingual CLIR is based on 1) word normalisation in indexing, 2)
stop-word lists, 3) normalisation of topic words, 4) splitting of compounds, 5) recognition of the right
components, 6) handling of non-translated words 7) phrase composition of compounds in the target language, 8)
bilingual dictionaries and 9) structured queries. For structuring of queries see
An overview of the new UTACLIR process is in Figure 1.
Language Query
Run: InQuery Matching
A new process for dictionary look-up and translation of compound words A new process for matching proper names and other non-translatable words New ways of using stop lists Normalisation of dictionary output
Unified process (Swedish and German) Dictionary look-up and translation of compound words. In the present process, normalised compound words, in case they are not stop words, are first attempted to look up in the dictionary (see Figure 2a). If a translation, or a set of translations, is available, it is likely to be the best alternative for the source word (a compound or noncompound). Such compounds often are non-compositional, i.e., a compound’s meaning may be quite different from the meanings of its components (e.g., strawberry). If the translation is a phrase, it will be handled as a phrase in the subsequent phases.
Cw
Compound words that do not translate are split into their components (see Figure 2b). For Swedish and German,
all consecutive component pairs are formed and translated (if possible). For example, for a four-component
compound a-b-c-d, the component pairs of a-b, b-c, and c-d are formed. Then these formed component pairs are
looked up in the dictionary. In the case of several translations, the equivalents are used as synonyms. If the
normalised component did not translate, it was modified by using a fogemorpheme algorithm before a new
translation attempt. If it still did not translate, then n-gram method (described below) was used for retrieving the
set of six most similar index terms with respect to the component. All combinations of the translation equivalents
are formed for the query. The rationale behind this method is that for a multi-component compound word it is
hard to know which consecutive components form common established compounds contained in the dictionary.
For German and Swedish compounds, we applied the fogemorpheme algorithm as
FM
Fogemorphemes
Componets
Untranslatable words. In the present process, proper names and other untranslatable words are handled by an
advanced n-gram method (see Figure 4). The method is language independent and is described
In the new process, stop word lists are used in a different way than last year. The new stop lists are used after the normalisation of words to base forms. Thus we do not have to include inflected word forms onto the lists. Some modifications were done to last year’s lists. Owing to the change in the process, only base form words were added onto the lists. This is important when dealing with highly inflectional source languages. Stop lists are not used for the target language query in the Swedish-English and the German-English processes. Normalisation of dictionary output. Dictionary output can include phrases and words in inflected forms. These do not match the normalised index terms. Therefore, because index terms were normalised, dictionary output words also were normalised.
Unified process. Our aim has been to unify the process for all source languages as much as possible. Language dependent features that have to be added to the process are included, but the initial process is unified. We are trying to develop parameters for the query construction phase, including possibilities to change operators, ways to handle components of compounds in the final query. In this year CLEF tests we have tried to unify the Swedish – English and German – English processes, however the process is not yet completely finished. Therefore there are minor differences in the programming solutions for proximity operators. The Finnish process is not yet adapted.
The runs
We participated
The main resources
Motcom Swedish – English translation dictionary (60.000 entries) by Kielikone Motcom Finnish – English translation dictionary (110.000 entries) by Kielikone Oxford Duden German – English translation dictionary (260.000 entries) Morphological analysers: SWETWOL, FINTWOL, GERTWOL and ENGTWOL by Lingsoft Inquery retrieval system German – English translation processes For German – English we have tested two types of dictionaries (two runs).
Using the Duden German-English dictionary (260.000 words) two translation tables for the 50 CLEF topics were created. The first one included all translations from the dictionary. The second translation table contained the same data, except that all direct translations of compounds were excluded. The construction of the GermanEnglish translation table was a separate process analysed by a human, following strict syntactic rules for selecting strings from the PC screen. German – English process could not be automated because of interface problems and colour fonts used in the Duden dictionary. However, the translation tables were used automatically.
The results Testrun TAYfinstr (Finnish structured) TAYswestr (Swedish structured) TAYgerstr (German structured) TAYgershort (German/short structured) The results of the four UTACLIR runs are presented below and in Figures 5 and 6.
Average precision (CLEF 2001) 0,3894 0,3769 0,3474 0,3054 (CLEF 2000) 0,2275 0,2540 0,2665 -------
SWEstr FINstr GERstr
GERshort 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9
Recall 1
0,7 0,6 0,5 iino0,4 s c reP0,3 0,2 0,1 0
Interpolated Recall-Precision Averages 0,45 0,4 0,35 0,3 n io0,25 s i ec 0,2 r P0,15 0,1 0,05 0
Document cut-off value
SWEstr FINstr GERstr GERshort 5 10 15 20 30
Generally the results for all the four runs are good. Improvements are considerable from the last year. Average precision for all the queries shows clear improvements, but there is still great variation in the performance of single queries. Some queries perform exceedingly well getting high scores, but some fail to retrieve relevant documents. This holds for all language pairs.
After last year CLEF evaluation we identified 1) problems with the translation of proper names (especially true for Finnish) 2) problems with untranslated components in the compound handling process 3) the handling of compounds containing more than two components was not satisfactory 4) normalisation is needed for the dictionary translation in order to match the normalised index of the database.
This year we focused on compound words and proper names. Both are important and common in the source languages and in this year’s topics. Therefore the way they are handled considerably affects the results. The new methods for compounds and proper names account for the improvements we achieved this year. However, part of the improvement may be due to the higher number of judged queries.
The choice of using the n-gram technique for all unidentified words was a success. This especially holds for Finnish, in which proper names quite often appear in inflected forms. However, the use of the n-gram technique also increased noise in the form of many nonsense words that were added to the final query. This particularly holds for Swedish and German where all untranslated single words and compound components were handled by the n-gram technique. If the untranslated word was not a proper name, nonsense words were added to the query by the n-gram technique. The Finnish process was different in this aspect since n-gram matching was not used as part of compound processing.
The new compound process was effective for some topics but failed for some topics. The Finnish run did not suffer from nonsense words generated from compounds which increased noise in conjunction with the n-gram technique. On the other hand, because we used last year’s process for Finnish (except for the proximity operator), compound components were not translated in all cases. This also had negative effects. The fogemorpheme algorithm seemed to improve the translations of compound components for both Swedish and German.
The test with two dictionaries for the German runs gives an indication that the new features in the UTACLIR process work well also with a limited dictionary. On the other hand the advantage of a direct translation of compounds is inevitable. Our method for handling compounds works as a good and necessary complement, since no dictionary even a comprehensive one holds entries for all compounds. Compound splitting is needed in several queries in the TAYgerstr-run where a comprehensive dictionary is used. The queries in the TAYgershortrun become very long since all compounds were split into their components. When all the alternative translations for the components are combined to a phrase in the target language query, the number of combinations may be high. Nonsense combinations also occurred quite frequently. On the other hand, generally the process can be said to work as expected because of the relevant combinations.
Acknowledgments ENGTWOL (Morphological Transducer Lexicon Description of English): Copyright (c) 1989-1992 Arto Voutilainen and Juha Heikkilä.
FINTWOL (Morphological Description of Finnish): Copyright (c) Kimmo Koskenniemi and Lingsoft Oy. 19831993.
GERTWOL (Morphological Transducer Lexicon Description of German): Copyright (c) 1997 Kimmo Koskenniemi and Lingsoft, Inc.
SWETWOL (Morphological Transducer Lexicon Description of Swedish): Copyright (c) 1998 Fred Karlsson and Lingsoft, Inc.
TWOL-R (Run-time Two-Level Program): Copyright (c) Kimmo Koskenniemi and Lingsoft Oy. 1983-1992. MOT Dictionary Software was used for automatic word-by-word translations. Copyright (c) 1998 Kielikone Oy, Finland.
The Inquery search engine was provided by the Center for Intelligent Information Retrieval at the University of Massachusetts. This research is part of the research project Query structures and dictionaries as tools in conceptbased and cross-lingual information retrieval funded by the Academy of Finland (Research Projects 44703; 49157).